DEV Community: Pradeep K M

Stop Deleting "Useless" Features: Why Univariate AUC is Lying to You

Pradeep K M — Tue, 13 Oct 2026 11:52:49 +0000

In high-stakes domains like BFSI (Banking, Financial Services, and Insurance) and Healthcare, the most valuable predictive signals are often hiding in plain sight.

Many data teams follow a standard pruning pipeline: calculate the univariate AUC for every feature, and if a feature scores near 0.5 (random noise), it gets tossed.

This decision is quietly killing your model's performance.

The Linear Trap vs. The Tree Power
The problem is a confusion between marginal and conditional effects.

The Linear Trap: In a Logistic Regression model, every feature gets a fixed coefficient (
β
β). It measures the effect of that feature in a vacuum. If "Age" has no individual linear relationship with the target across the whole dataset, it gets a high p-value and is zeroed out.

The Tree Power: Gradient Boosting Decision Trees (GBDTs) like XGBoost or CatBoost operate on conditional logic. They don't ask, "Does Age matter?" They ask, "Does Age matter given that Income is low and Education is X?"

In sparse datasets, a feature might be globally irrelevant but locally critical.

Proving it with Code
Here is a simplified demonstration of how a feature can be "useless" on its own but essential when paired with another variable.

import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score
from xgboost import XGBClassifier
import shap
import matplotlib.pyplot as plt

# 1. Create a synthetic dataset
np.random.seed(42)
n_samples = 1000

# Feature A: Pure noise
feat_a = np.random.randn(n_samples)

# Feature B: Also looks like noise globally
feat_b = np.random.randn(n_samples)

# Target: Only depends on the INTERACTION of A and B
# Target is 1 ONLY if both A and B are positive (a non-linear pocket)
target = ((feat_a > 0) & (feat_b > 0)).astype(int)

df = pd.DataFrame({'Feature_A': feat_a, 'Feature_B': feat_b, 'Target': target})

# 2. The "Univariate Check" (The mistake)
auc_a = roc_auc_score(df['Target'], df['Feature_A'])
auc_b = roc_auc_score(df['Target'], df['Feature_B'])

print(f"Univariate AUC Feature A: {auc_a:.3f}") # ~0.50
print(f"Univariate AUC Feature B: {auc_b:.3f}") # ~0.50
print("Result: A data scientist might delete both features here! ❌")

# 3. The Boosting Approach
X = df[['Feature_A', 'Feature_B']]
y = df['Target']

model = XGBClassifier(max_depth=3, n_estimators=100)
model.fit(X, y)

# 4. Explain with SHAP
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)

print("\nSHAP analysis reveals that both features are critical for the interaction.")
shap.summary_plot(shap_values, X)

What is SHAP actually telling us?
When you see a feature with a weak univariate score but a high SHAP value, the model is telling you:

"This variable cannot carry the model alone, but across the forest's decision paths, knowing this variable helped me carve out a high-precision pocket of truth."

The Takeaway for Data Leaders
If you are working with complex, multi-dimensional data in Healthcare or Finance, remember: Global irrelevance
≠

= Local uselessness.

Respect Interactions: Healthcare and BFSI data thrive on combinations, not solo acts.
Audit your Pruning: Before deleting a feature based on a p-value or univariate AUC, check its interaction strength via a tree-based model.
Trust Non-Linear Architectures: Let your boosting models exploit the conditional depth they were designed for.

About the Author:
I specialize in building high-precision predictive models for BFSI and Healthcare, focusing on extracting signal from sparse, non-linear datasets through advanced feature engineering and XAI. I'm currently open to new opportunities where I can help teams turn "noise" into actionable insights.

NSE-MCP: A Step Toward AI-Ready Data, or Just Playing Catch-up? 📈🤖

Pradeep K M — Sat, 03 Oct 2026 17:05:44 +0000

As someone who has spent a significant amount of time working with NSE (National Stock Exchange of India) data, I was intrigued to see the launch of the NSE-MCP (Model Context Protocol) initiative.

For those who aren't aware, MCP is designed to standardize how LLMs and AI agents connect to data sources. Instead of writing a custom integration for every single API endpoint, MCP provides a "plug-and-play" interface.

The Good Part ✅

The initiative opens up two critical paths:

Historical Analysis: Access to Bhavcopy data from the last 5 years.
Near Real-Time Monitoring: A 1–3 minute delayed live feed (CM Market Live).

From a DX (Developer Experience) perspective, moving toward a standardized protocol is a huge win. It reduces the boilerplate code required to feed financial context into an LLM and makes the data more "discoverable" for AI agents.

The "Real Talk" 🚩

However, looking at this from an SME perspective... the NSE is late to the party.

If you've ever tried to build a production-grade fintech app in India, you know the struggle. For years, the community has been forced to build complex middleware, custom scrapers, and fragile wrappers just to get institutional data into a usable format for modern AI workflows.

While the architecture of MCP is the right choice, the timing feels reactive. The LLM explosion happened years ago; providing a standardized context protocol now is like installing a high-speed rail line after everyone has already bought electric scooters to get around the traffic.

The Verdict ⚖️

The most disappointing part? The current limitation to "educational and informational purposes only."

For a developer, "educational" is code for "don't try to build a real business on this yet." To truly lead the fintech space, the NSE needs to move beyond the sandbox and provide a production-ready, high-throughput infrastructure that doesn't shy away from commercial utility.

Is it a step forward? Yes.
Is it a lead? No. It's a recovery.

I’m curious to hear from other devs and data engineers:

Are you already using MCP in your stack?
Do you think this will actually reduce the friction of working with Indian market data?
Or are you sticking with your custom pipelines?

Let's discuss in the comments! 👇

fintech #ai #mcp #nse #dataengineering #llm #india #tradingbot

{
  "mcpServers": {
    "nse-bhavcopy": {
      "url": "https://mcp.nseindia.in/bhavcopy/cm/mcp",
      "transport": "streamable-http"
    },
    "cm-market": {
      "url": "https://mcp.nseindia.in/cmmkt/mcp",
      "transport": "streamable-http"
    }
  }
}

import asyncio
import json
import polars as pl
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client

async def fetch_reliance_quote_clean():
    url = "https://mcp.nseindia.in/cmmkt/mcp"
    print(f"Connecting to NSE MCP server at {url}...")

    async with streamablehttp_client(url) as (read, write, session_id):
        async with ClientSession(read, write) as session:
            await session.initialize()

            target_tool = "cm_get_stock_quote"
            arguments = {"symbol": "RELIANCE"}

            print(f"Calling tool '{target_tool}' with arguments: {arguments}")
            result = await session.call_tool(target_tool, arguments=arguments)

            # Extract raw text content
            raw_text = None
            for content in result.content:
                if content.type == "text":
                    raw_text = content.text
                    break

            if not raw_text:
                print("Error: No text data returned from the MCP tool.")
                return

            try:
                parsed_json = json.loads(raw_text)

                # If the response wraps data in a dictionary, structure it for Polars
                if isinstance(parsed_json, dict):
                    # Extract fields if they are nested inside 'stock' or similar keys
                    data_list = [parsed_json]
                else:
                    data_list = parsed_json

                df = pl.DataFrame(data_list)

                # Check if 'stock' column exists as a struct and unpack it safely
                if "stock" in df.columns:
                    df = df.unnest("stock")

                print("\n--- Unpacked Polars DataFrame ---")
                print(df)

                # Export to CSV (now flattened successfully)
                output_filename = "reliance_live_quote_flat.csv"
                df.write_csv(output_filename)
                print(f"\nSuccessfully exported flattened data to {output_filename}")

                return df

            except json.JSONDecodeError:
                print("Failed to parse response text as JSON. Raw output:")
                print(raw_text)

# Run directly in your interactive cell
await fetch_reliance_quote_clean()

Bridging the Knowledge Gap: Why General LLMs Fail at HEOR and How to Fix it with RAG 🏥🤖

Pradeep K M — Mon, 28 Sep 2026 10:18:07 +0000

The ambition for AI in European healthcare is sky-high. Policymakers see AI as the key to optimizing health budgets and patient outcomes. However, there is a critical bottleneck: General-purpose LLMs are fundamentally mismatched with the reality of Health Economics and Outcomes Research (HEOR).
The Problem: The "Knowledge Gap"

General AI is trained on the public web. But Real-World Evidence (RWE) is:

Siloed & Protected: GDPR prevents "scraping" sensitive patient trajectories.
Noisy: EHRs are fragmented and inconsistently coded.
Logic-Heavy: Calculating a QALY (Quality-Adjusted Life Year) requires longitudinal reasoning, not just probabilistic word prediction.

In short: General AI simulates the language of health economics without possessing the underlying data-driven logic.
The Solution: Moving from General AI to RAG

To bridge this gap, we must stop relying on the model's internal weights and start using Retrieval-Augmented Generation (RAG). Instead of asking the AI to "remember" a medical fact, we provide it with a secure, retrieved slice of actual RWE data to analyze in real-time.

Below is a conceptual Python implementation using LangChain and ChromaDB to show how we can ground an LLM in specific, secure medical documentation.

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

1. Setup: Use a secure, local vector store to avoid data leakage

In a real HEOR scenario, this would be connected to a

Federated Learning node or a secure hospital vault.

os.environ["OPENAI_API_KEY"] = "your-api-key"

def setup_heor_rag_pipeline(document_path):
# Load secure RWE/HEOR documentation
loader = PyPDFLoader(document_path)
documents = loader.load()

# Create embeddings - converting medical text into vectors
embeddings = OpenAIEmbeddings()

# Store in a local vector database (ChromaDB)
# This ensures the data stays under our control, not in the model's training set
vectorstore = Chroma.from_documents(
    documents=documents, 
    embedding=embeddings, 
    persist_directory="./heor_secure_vault"
)

return vectorstore

2. The "Reasoning" Layer

We use a custom prompt to force the AI to act as a Health Economist

template = """
You are a specialized HEOR Expert. Use the following pieces of retrieved
Real-World Evidence (RWE) to answer the user's question.
If the evidence does not contain the answer, state that the data is insufficient.
Do not simulate or hallucinate figures.

Context: {context}
Question: {question}

Expert Analysis:"""

QA_CHAIN_PROMPT = PromptTemplate(
input_variables=["context", "question"],
template=template,
)

def analyze_health_economics(vectorstore, query):
llm = ChatOpenAI(model_name="gpt-4", temperature=0) # Low temp for precision

qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=vectorstore.as_retriever(),
    chain_type_kwargs={"prompt": QA_CHAIN_PROMPT}
)

return qa_chain.invoke(query)

--- Execution ---

if name == "main":
# Assume 'clinical_trial_rwe.pdf' contains granular patient trajectory data
vault = setup_heor_rag_pipeline("clinical_trial_rwe.pdf")

question = "Based on the provided RWE, what is the incremental cost-effectiveness ratio (ICER) for Therapy X compared to the standard of care?"
result = analyze_health_economics(vault, question)

print(f"Analysis: {result['result']}")

Why the "Newest" Healthcare Data is Often the Worst for ML

Pradeep K M — Sat, 26 Sep 2026 13:29:21 +0000

If you are building machine learning models using healthcare claims data, your biggest enemy isn't usually the algorithm—it's the claims lag.

When working with real-world data (RWD), there is a common temptation to use the most recent data available to keep models "current." In reality, using the most recent 60 days of claims data is often a recipe for model failure and skewed results.

The Problem: The "Incomplete Picture"

Healthcare claims aren't instantaneous. Between provider submissions, insurance adjudication, corrections, and deduplication, there is a significant time gap between a clinical event and a finalized claim.

If you train a model on data from "yesterday," you aren't seeing the full truth. You are seeing a filtered version of the truth where only the fastest-reporting providers are represented. This creates a massive late-claim bias.

The Solution: The X–60 Day Analytical Cut-off

To solve this, I implement an analytical cut-off. Instead of using the data extraction date (X), I use:

T = X − 60 days

By ignoring the most recent 60 days, we allow the data to "mature." This approach provides several critical advantages for ML stability:

Reducing Undercounting: Ensures that a patient isn't flagged as "inactive" simply because their claim is still being processed.
Preventing Data Leakage: Prevents the model from picking up on temporal artifacts that wouldn't exist in a real-world deployment.
Consistency: Creates a stable baseline for train/test splits, ensuring that the "completeness" of the data is uniform across both sets.

Warning: 60 Days is Not a Magic Number

A 60-day lag is a starting point, not a universal law. The required "maturity window" varies wildly depending on the data source:

Data Type	Typical Maturity Lag
Pharmacy Claims	Short (30–60 days)
Hospitalizations	Medium (60–120 days)
Mortality/Outcomes	Long (180+ days)

How to Determine Your Own Cut-off

If you're unsure what lag to use, don't guess. Run a Completeness Test:

Compare the volume and characteristics of your data at 30, 60, 90, and 180-day intervals. Plot the "capture rate" over time. Once the curve flattens (reaches an asymptote), you've found your optimal analytical cut-off.

The Golden Rule of Healthcare ML: Use the newest sufficiently mature data—not simply the newest data available.