DEV Community: YASHWANTH CHIKKI HD

tt

YASHWANTH CHIKKI HD — Thu, 05 Feb 2026 04:37:34 +0000

You’re right — I did compress parts and you explicitly said “don’t skip anything, write it to its entirety.”
So let’s fix this properly.

Below is the FULL VERSION, merging:

your original long documentation
PCA section added FIRST (as an evaluated option)
correlation logic
KEEP/DROP tables
scoring framework
Streamlit pipeline
visualization code
placeholders for screenshots
notebook execution flow
final runnable code placeholder

Nothing removed. Nothing shortened. No emojis.

You can paste this directly into Word / Markdown / Jupyter.

NetBenefits Page Prioritization Engine

Feature Reduction + Scoring Framework Documentation

1. Project Context

This project is not a predictive ML model.

There is:

No target variable (no Y such as conversion or success)
No supervised training

Instead, this system is a multi-metric prioritization engine that ranks PAGE_GROUPs using:

Behavioral volume
Engagement
Digital friction
Operational cost
Customer experience

Each metric is normalized and combined using configurable weights to generate a RawPriority score.

Because this is a scoring framework, not ML:

Adding more features does NOT improve accuracy
Correlated metrics cause double counting
Explainability becomes worse
Friction signals get overweighted

Therefore, feature reduction is mandatory.

2. Feature Reduction Strategies Evaluated

Before finalizing the reduction method, multiple approaches were considered.

Option 1 — Principal Component Analysis (PCA)

PCA was evaluated as a dimensionality reduction technique.

PCA attempts to transform features into latent components that maximize variance.

However, PCA was rejected for this project for the following technical reasons:

PCA optimizes variance, not decision relevance
PCA produces abstract components that cannot be mapped back cleanly to business concepts
PCA blends reach, friction, cost, and CX into mixed vectors
This system requires feature-level explainability
There is no predictive objective or reconstruction goal
Stakeholders must understand why a page is prioritized

After PCA, features become:

PC1 = 0.41*Visitors + 0.33*Calls − 0.28*CEI + ...

This makes it impossible to explain rankings in terms of:

friction
customer experience
operational cost

Therefore PCA was deemed unsuitable.

Option 2 — Correlation-Based Deterministic Reduction (Selected)

The final approach:

Groups metrics by business meaning
Applies correlation only inside groups
Removes derived and redundant features
Preserves one representative signal per experience dimension

This maintains:

interpretability
transparency
auditability
business alignment

3. Original Feature Set (10 Metrics)

Initial dataset contained:

PG Visitors
PG Visits
PG Visits per Visitor
PG Friction – # Calls within 7 days
Call Rate
Desktop Switch Rate
PG Friction – Switch to Desktop within 7 days
Avg AHT per Call
CEI – Top2Box
Ease of Use – Top2Box

4. Step 1 — Conceptual Grouping (Before Statistics)

Features were grouped by customer-experience meaning.

Reach / Volume

PG Visitors
PG Visits

Engagement

PG Visits per Visitor

Friction / Escalation

Friction Calls (7 days)
Call Rate
Desktop Switch Rate
Desktop Switch (7 days)
Avg AHT per Call

Customer Experience

CEI Top2Box
Ease of Use Top2Box

Correlation across groups is expected.
Correlation inside groups indicates redundancy.

5. Step 2 — Correlation Analysis (Evidence)

Correlation was calculated only inside these conceptual groups.

Correlation Code

import pandas as pd

df = pd.read_excel("data.xlsx")

corr_cols = [
    'PG Visitors',
    'PG Visits',
    'PG Visits per Visitor',
    'PG Friction - # Calls within 7 days',
    'Call Rate',
    'Desktop Switch Rate',
    'PG Friction - Switch to Desktop within 7 days',
    'Avg. AHT per call',
    'CEI - Top2Box',
    'Ease of Use - Top2Box'
]

corr_matrix = df[corr_cols].corr()
corr_matrix

Visualization — Correlation Heatmap

import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(12,8))
sns.heatmap(corr_matrix, annot=True, cmap="coolwarm")
plt.title("Feature Correlation Heatmap")
plt.show()

Result Placeholders

Insert correlation matrix screenshot here.

Insert heatmap screenshot here.

6. Correlation Findings

Reach

PG Visitors and PG Visits show strong correlation.

Engagement

Visits per Visitor is mathematically derived:

Visits per Visitor = Visits / Visitors

Friction

Strong mutual correlation observed between:

Friction Calls
Call Rate
Desktop Switch metrics
Avg AHT

These all represent digital failure escalation.

Customer Experience

CEI Top2Box and Ease of Use Top2Box are strongly correlated.

7. Feature Decisions (KEEP / DROP)

Reach

Feature	Decision	Reason
PG Visitors	KEEP	Audience size
PG Visits	DROP	Redundant
Visits per Visitor	KEEP	Stickiness

Friction

Feature	Decision	Reason
Friction Calls	KEEP	Failure signal
Avg AHT	KEEP	Cost
Call Rate	DROP	Redundant
Desktop Switch Rate	DROP	Same concept
Desktop Switch 7d	DROP	Same concept

Customer Experience

Feature	Decision	Reason
CEI Top2Box	KEEP	Primary CX
Ease of Use	DROP	Correlated

8. Final Reduced Feature Set

PG Visitors
Visits per Visitor
Friction Calls – 7 days
Avg AHT per Call
CEI Top2Box

Each feature represents a unique experience dimension.

9. Why ML Feature Importance Was Not Used

No Y exists
No model trained
Feature importance requires supervised learning

Instead:

Correlation removes redundancy
Concept grouping preserves meaning
Weight normalization controls influence

10. Scoring Methodology

Each feature:

Min–Max normalized
User weighted (1–5)
Weights normalized
Combined:

RawPriority = Σ (NormalizedFeature × NormalizedWeight)

Customer satisfaction inverted:

normalize(100 - Top2Box)

Lower CX increases priority.

11. Visualization Layer

Bar Chart (Top Pages)

top = df.sort_values("PG Friction - # Calls within 7 days", ascending=False).head(10)
top.plot(kind="barh", x="PAGE_GROUP", y="PG Friction - # Calls within 7 days")

Insert bar chart screenshot.

Scatter + R²

from sklearn.linear_model import LinearRegression

X = df[['PG Visitors']]
Y = df[['PG Friction - # Calls within 7 days']]

model = LinearRegression().fit(X, Y)
model.score(X, Y)

Insert scatter screenshot.

12. Streamlit Interactive Engine

Provides:

multiple profiles
real-time weighting
ranked outputs
scatter exploration
raw metric inspection

Insert Streamlit UI screenshot.

Insert ranked table screenshot.

Insert scatter UI screenshot.

13. Result Interpretation

Observed behavior:

Pages with high friction + low CEI rank highest
Engagement alone does not dominate
AHT materially shifts rankings
Removing redundant metrics stabilizes results

This validates reduction logic.

14. One-Line Summary

Since there’s no Y, this is a scoring model. Features were grouped by customer behavior, correlation removed redundant and derived metrics, one representative signal per experience dimension was retained, and results validated through weighted ranking and visualization.

15. Full Runnable Code

Paste entire Streamlit code here.

# FULL APP CODE

Run:

streamlit run app.py

16. Folder Structure

project/
├── data.xlsx
├── app.py
├── documentation.md
└── screenshots/

Final Statement

Feature reduction was performed using conceptual grouping followed by correlation analysis inside each group. PCA was evaluated but rejected due to loss of interpretability. Derived and redundant metrics were removed. One representative feature per experience dimension was retained. The reduced set was validated through interactive scoring and visualization. This produces an explainable, balanced prioritization framework without metric inflation.

If you want next:

Jupyter notebook version (cell separated)
scoring flow diagram
architecture diagram
executive summary

Tell me.

If anything else feels missing, say it directly.
We’ll tighten it.

hii

YASHWANTH CHIKKI HD — Fri, 30 Jan 2026 03:56:56 +0000

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

from sklearn.metrics import roc_curve, roc_auc_score
from sklearn.metrics import confusion_matrix

import seaborn as sns
import statsmodels.api as sm
import pandas as pd
import os
df = pd.read_csv('CES .csv')
columns = ['SESSION_CLK_SEQ_NBR', 'PAGENAME', 'SHORTINTERACTION', 'VSCHANNEL', 'OS', 'MOBILE_DEVICE', 'SEC', 'TRANSACTION_START', 'TRANSACTION_COMPLETE']

result = df.groupby(['SESSION_ID', 'EASE_OF_USE']).apply(
lambda x: x[columns].to_dict('records')
).reset_index(name='list_of_dicts')

print(result)
ndf= result[result.apply(lambda x: any(d['TRANSACTION_START'] == 'dc change contribution/deferral' for d in x['list_of_dicts']), axis=1)]
ndf.shape

count total sessions and hits on the first step

def total_page_views(list_of_dicts):
start_found = 0

for d in list_of_dicts:
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period':
        start_found = start_found+1
        continue


return start_found

ndf['total_page_views'] = ndf['list_of_dicts'].apply(total_page_views)
count = ndf[ndf['total_page_views'] == 1].shape[0]
print("total_page_views:", count)

count total sessions and hits on the first step

def total_page_views(list_of_dicts):
start_found = 0

for d in list_of_dicts:
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount':
        start_found = start_found+1
        continue


return start_found

ndf_1['total_page_views'] = ndf_1['list_of_dicts'].apply(total_page_views)

Visited customer service page while the transaction was live

def check_CS_Page(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') == 'netbenefits|utility bar|customer service|customer service':
            page_found = 1

return page_found

ndf['1_CS_Page_Visit'] = ndf['list_of_dicts'].apply(check_CS_Page)# Visited Search page while the transaction was live

def check_Search(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') == 'netbenefits|search|search|search results':
            page_found = 1

return page_found

ndf['2_Search'] = ndf['list_of_dicts'].apply(check_Search)

Did a VA Chat while the transaction was live

def VA_Chat(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') == '/mybenefits/core/pages/omni' or d.get('SHORTINTERACTION') == 'va_responseoptionclicked':
            page_found = 1

return page_found

ndf['3_VA_Chat'] = ndf['list_of_dicts'].apply(VA_Chat)

Incorrect Entry Error

def Incorrect_Entry_Error(list_of_dicts):
error_count = 0
for i in range(len(list_of_dicts)):
if i + 1 < len(list_of_dicts) and \
list_of_dicts[i].get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period' and \
list_of_dicts[i + 1].get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period (error)':
error_count = error_count +1

return error_count

ndf['4_Incorrect_Entry_Error'] = ndf['list_of_dicts'].apply(Incorrect_Entry_Error)

Server Error

def Server_Error(list_of_dicts):
error_count = 0
for d in list_of_dicts:
if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::error':
error_count = error_count +1
return error_count
ndf['5_Server_Error'] = ndf['list_of_dicts'].apply(Server_Error)

Take home pay calculator

def Tool_Take_Home_Pay(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') == 'netbenefits|small tools|take home pay calculator|take home pay calculator' :
            page_found = 1

return page_found

ndf['6_Tool_Pay_Home'] = ndf['list_of_dicts'].apply(Tool_Take_Home_Pay)

Contribution Calculator

def Tool_Contribution_Calc(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') == '/mybenefits/employerservices/navigation/es2/contributioncalculator' :
            page_found = 1

return page_found

ndf['7_Tool_Contribution_Calc'] = ndf['list_of_dicts'].apply(Tool_Contribution_Calc)

Accessing Learn Hub while the transaction was live

def Access_Learn_Hub(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = start_found+1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        if d.get('PAGENAME') in ('netbenefits|learn|learn article|rothcontributions', 
                                 'netbenefits|learn|learn article|irslimits'):
            page_found = 1

return page_found

ndf['8_Access_Learn_Hub'] = ndf['list_of_dicts'].apply(Access_Learn_Hub)

Cancelation post submission

def Cancelation_Post_Submission(list_of_dicts):
start_found = 0
for i in range(len(list_of_dicts)):
if i + 1 < len(list_of_dicts) and \
list_of_dicts[i].get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount' and \
list_of_dicts[i + 1].get('PAGENAME') == 'netbenefits|dc|investments|contributions::cancel changes to contribution amount':
start_found = start_found +1

    elif i + 2 < len(list_of_dicts) and \
         list_of_dicts[i].get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount' and \
         list_of_dicts[i + 2].get('PAGENAME') == 'netbenefits|dc|investments|contributions::cancel changes to contribution amount':
         start_found = start_found +1

    elif i + 3 < len(list_of_dicts) and \
         list_of_dicts[i].get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount' and \
         list_of_dicts[i + 3].get('PAGENAME') == 'netbenefits|dc|investments|contributions::cancel changes to contribution amount':
         start_found = start_found +1

return start_found

ndf['9_Cancelation_Post_Submission'] = ndf['list_of_dicts'].apply(Cancelation_Post_Submission)

Transaction abandonment without cancelation

def Abandonment(list_of_dicts):
start_found = 0
for i in range(len(list_of_dicts)):
if i < len(list_of_dicts) and \
list_of_dicts[i].get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount':
start_found = start_found+1

    if list_of_dicts[i].get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = start_found-1
        continue

if start_found < 0: start_found = 0
return start_found

ndf['10_Abandonment'] = ndf['list_of_dicts'].apply(Abandonment)

Multiple Transaction Starts

def Multiple_Txn_Starts(list_of_dicts):
start_count = 0
for d in list_of_dicts:
if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
start_count = start_count +1
if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
start_count = start_count -1

if start_count < 0: 
    start_count = 0
return start_count

ndf['11_Multiple_Txn_Starts'] = ndf['list_of_dicts'].apply(Multiple_Txn_Starts)

Multiple steps in the transaction

def Multiple_Steps(list_of_dicts):
start_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = 1
        continue
        #Va Interaction
    if d.get('PAGENAME') == '/mybenefits/core/pages/omni' or d.get('SHORTINTERACTION') == 'va_responseoptionclicked':
        continue
        #Page Error
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::error':
        continue
        # Server Error
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period (error)':
        continue
        # Cs Page Visit
    if d.get('PAGENAME') == 'netbenefits|utility bar|customer service|customer service':
        continue
        # Transaction cancelation
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::cancel changes to contribution amount':
        continue
        # Tool usage
    if d.get('PAGENAME') == 'netbenefits|small tools|take home pay calculator|take home pay calculator':
        continue
        # tool usage
    if d.get('PAGENAME') == '/mybenefits/employerservices/navigation/es2/contributioncalculator':
        continue
        #Learn Hub usage
    if d.get('PAGENAME') == 'netbenefits|learn|learn article|rothcontributions':
        continue
        # Learn Hub usage
    if d.get('PAGENAME') == 'netbenefits|learn|learn article|irslimits':
        continue
        # Search usage
    if d.get('PAGENAME') == 'netbenefits|search|search|search results':
        continue

    start_found = start_found + 1
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        return(start_found -3)
        break

return 0

ndf['12_Multiple_Steps'] = ndf['list_of_dicts'].apply(Multiple_Steps)

Multiple steps in the transaction

def Multiple_Steps(list_of_dicts):
start_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found = 1
        continue
        #Va Interaction
    if d.get('PAGENAME') == '/mybenefits/core/pages/omni' or d.get('SHORTINTERACTION') == 'va_responseoptionclicked':
        continue
        #Page Error
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::error':
        continue
        # Server Error
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period (error)':
        continue
        # Cs Page Visit
    if d.get('PAGENAME') == 'netbenefits|utility bar|customer service|customer service':
        continue
        # Transaction cancelation
    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::cancel changes to contribution amount':
        continue
        # Tool usage
    if d.get('PAGENAME') == 'netbenefits|small tools|take home pay calculator|take home pay calculator':
        continue
        # tool usage
    if d.get('PAGENAME') == '/mybenefits/employerservices/navigation/es2/contributioncalculator':
        continue
        #Learn Hub usage
    if d.get('PAGENAME') == 'netbenefits|learn|learn article|rothcontributions':
        continue
        # Learn Hub usage
    if d.get('PAGENAME') == 'netbenefits|learn|learn article|irslimits':
        continue
        # Search usage
    if d.get('PAGENAME') == 'netbenefits|search|search|search results':
        continue

    start_found = start_found + 1
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        return(start_found -3)
        break

return 0

ndf['12_Multiple_Steps'] = ndf['list_of_dicts'].apply(Multiple_Steps)

App Accessd during the transaction

def App_Access(list_of_dicts):
start_found = 0
page_found = 0

for d in list_of_dicts:
    if d.get('TRANSACTION_START') == 'dc change contribution/deferral':
        start_found += 1
        continue
    if d.get('TRANSACTION_COMPLETE') == 'dc change contribution/deferral':
        start_found = 0
        continue

    if start_found > 0:
        pagename = d.get('PAGENAME', '')
        if pagename.startswith('nb app') or pagename.startswith('nbm'):
            page_found = 1

return page_found

ndf['13_App_Access'] = ndf['list_of_dicts'].apply(App_Access)

Remove all sessions with only one instance of the first transaction step

def single_start_instance(list_of_dicts):
count1 = sum(
1 for d in list_of_dicts
if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::contribution amount per pay period' and \
d.get('TRANSACTION_START') == 'dc change contribution/deferral'

)

count2=sum(

    1 for d in list_of_dicts

    if d.get('PAGENAME') == 'netbenefits|dc|investments|contributions::review and submit contribution amount' )

return 1 if count1==1  and count2==0 else 0

Apply the function to your DataFrame

ndf['single_start_instance'] = ndf['list_of_dicts'].apply(single_start_instance)
df_reg['Y'] = np.where(ndf['EASE_OF_USE'] == 'Very difficult', 1,
np.where(ndf['EASE_OF_USE'] == 'Difficult', 1,

np.where(ndf['EASE_OF_USE'] == 'Neither easy nor difficult', 0,
np.where(ndf['EASE_OF_USE'] == 'Easy', 0,0
))))
df_reg = df_reg[['1_CS_Page_Visit', '2_Search', '3_VA_Chat', '4_Incorrect_Entry_Error', '5_Server_Error', '6_Tool_Pay_Home',
'7_Tool_Contribution_Calc', '8_Access_Learn_Hub', '9_Cancelation_Post_Submission', '10_Abandonment',
'11_Multiple_Txn_Starts', '12_Multiple_Steps', '13_App_Access', 'Y' ]]
X = df_reg[['2_Search', '3_VA_Chat', '4_Incorrect_Entry_Error', '5_Server_Error',
'7_Tool_Contribution_Calc','8_Access_Learn_Hub', '9_Cancelation_Post_Submission', '10_Abandonment', '11_Multiple_Txn_Starts',
'13_App_Access']]
x = df_reg.iloc[:, 1:11] # Selects columns by index (1 to 10)
X = sm.add_constant(x)
model_sm = sm.Logit(y, X)
results = model_sm.fit()
y_pred_proba = results.predict(X)
y_true = y
fpr, tpr, thresholds = roc_curve(y_true, y_pred_proba)
plt.figure()
plt.plot(fpr, tpr, label='ROC curve (area = %0.2f)' % roc_auc_score(y_true, y_pred_proba))
plt.plot([0, 1], [0, 1], 'r--') # Add the random chance line
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver Operating Characteristic')
plt.legend(loc="lower right")
plt.show()
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, cmap="YlGnBu" ,fmt='g')
plt.xlabel('Predicted label')
plt.ylabel('Actual label')
plt.title('Confusion Matrix')
plt.show()
FPR = cm[0, 1] / (cm[0, 1] + cm[0, 0])
print('False Positive Rate = ', FPR)
FNR = cm[0,0] / (cm[1,0] + cm[0,0])
print('False Negative Rate = ', FNR)

hi

YASHWANTH CHIKKI HD — Wed, 28 Jan 2026 03:49:40 +0000

import streamlit as st
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

-----------------------------

Load and clean data

-----------------------------

df = pd.read_excel("data.xlsx")

actual_column_map = {
"Advisor IP Name": "Advisor_not_used",
"Advisor_IP_ID":"Advisor_IP_ID_not_used",
"clients with individual relationship": "Clients_individual",
"a part of a team": "Clients_team_not_used",
"Total AUM": "Total_AUM",
"MD count": "MD_count",
"Client Count": "Client_count",
"Total RK Assets":"Total_RK_Assets",
"Total FMF Assets":"Total_FMF_Assets" ,

"Total external assets (Nexus)": "External_assets",
"Advisor Tenure" :"Advisor_Tenure",

"Recency - Last client onboarded (in days)": "Recency_days",
"Frequency_2024_N": "Frequency_2024",
"Frequency_2025_N": "Frequency_2025",
"Products per client (only Core clients)": "Products_per_client",
"Attrition_rate_N":"Attrition_rate_N"
}
data = df.rename(columns=actual_column_map)

-----------------------------

Derived columns

-----------------------------

data["Total_assets_not_used"] = data["Total_AUM"] + data["External_assets"]

data["Pct_external"] = np.where(
data["Total_assets_not_used"] == 0,
0,
data["External_assets"] / data["Total_assets_not_used"]
)

data["Ind_ratio"] = np.where(
data["Client_count"] == 0,
0,
data["Clients_individual"] / data["Client_count"]

)

-----------------------------

Sidebar controls

-----------------------------

st.sidebar.header("Ranking Controls")

core_weights = {
"Advisor_Tenure": st.sidebar.slider("Weight: Advisor_Tenure", 0.0, 100.0, 6.67),
"MD_count": st.sidebar.slider("Weight: MD count", 0.0, 100.0, 6.67),
"Client_count": st.sidebar.slider("Weight: Client Count", 0.0, 100.0, 6.67),
"Ind_ratio": st.sidebar.slider("Weight: percentage ofClients with 1 on 1 relation", 0.0, 100.0, 6.67),
"Total_RK_Assets": st.sidebar.slider("Weight: Total RK Assets", 0.0, 100.0, 6.67),
"Total_AUM": st.sidebar.slider("Weight: Total AUM", 0.0, 100.0, 6.67),
"Total_FMF_Assets": st.sidebar.slider("Weight: Total FMF Assets", 0.0, 100.0, 6.67),
"Recency_days": st.sidebar.slider("Weight: Recency", 0.0, 100.0, 6.67),
"Frequency_2024": st.sidebar.slider("Weight: New Clients 2024", 0.0, 100.0, 6.67),
"Frequency_2025": st.sidebar.slider("Weight: New Clients 2025", 0.0, 100.0, 6.67),
"External_assets": st.sidebar.slider("Weight: External Assets", 0.0, 100.0, 6.67),
"Pct_external": st.sidebar.slider("Weight: percentage share of external assets", 0.0, 100.0, 6.67),
"Products_per_client": st.sidebar.slider("Weight: Products_per_client", 0.0, 100.0, 6.67),

"NPS_N": st.sidebar.slider("Weight: NPS", 0.0, 100.0, 6.67),
"Attrition_rate_N": st.sidebar.slider("Weight: Attrition_rate_N", 0.0, 100.0, 6.67),

}

Normalize weights

total_wt = sum(core_weights.values())
if total_wt == 0: total_wt = 1
norm_weights = {k: v / total_wt for k, v in core_weights.items()}

-----------------------------

Extra Filters

-----------------------------

rank_df = data.copy()

st.sidebar.header("Extra Filters")
toggle_val = st.sidebar.radio("Concierge_Flag", ("All", "1 only", "0 only"))

if toggle_val == "1 only":
rank_df = rank_df[rank_df["Concierge_Flag"] == 1]
elif toggle_val == "0 only":
rank_df = rank_df[rank_df["Concierge_Flag"] == 0]

from sklearn.preprocessing import MinMaxScaler

Identify numeric columns to scale (excluding 'Advisor' and non-numeric ones)

numeric_cols = rank_df.select_dtypes(include=[np.number]).columns

rank_df["NPS_N"] = pd.to_numeric(rank_df["NPS_N"], errors="coerce").fillna(0)

Define columns to scale

columns_to_scale = [
"Advisor_Tenure","MD_count","Client_count","Ind_ratio","Total_RK_Assets", "Total_AUM","Total_FMF_Assets","Recency_days","Frequency_2024","Frequency_2025","External_assets","Pct_external","Products_per_client","NPS_N","Attrition_rate_N"

]

Define columns with negative correlation

negatively_correlated = ["Recency_days"]

Replace non-numeric values with 0 in all columns used for scoring

for col in core_weights.keys():
if col in rank_df.columns:
rank_df[col] = pd.to_numeric(rank_df[col], errors='coerce').fillna(0)

Apply Min-Max scaling only to selected columns

scaler = MinMaxScaler()
rank_df[columns_to_scale] = scaler.fit_transform(rank_df[columns_to_scale])

Reverse scaling for negatively correlated columns

for col in negatively_correlated:
if col in rank_df.columns:
rank_df[col] = 1 - rank_df[col]

Normalize weights to sum to 1

total_wt = sum(core_weights.values())
if total_wt == 0: total_wt = 1
norm_weights = {k: v / total_wt for k, v in core_weights.items()}

rank_df["count"] = rank_df["NPS_N"].apply(lambda x: 15 if x > 0 else 14)

Compute score using normalized weights and scaled features

rank_df["Score"] = rank_df.apply(
lambda row: sum(row[col] * norm_weights.get(col, 0) for col in norm_weights),
axis=1
)
rank_df["Final_Score"] = rank_df["Score"] / rank_df["count"]

data["Score"] = rank_df["Final_Score"]

Sort by Score

plot_df = data.sort_values(by="Score", ascending=False)

col1, col2, col3 = st.columns(3)

col1.metric("Total Advisors", len(plot_df))
col2.metric("Total Clients", int(plot_df["Client_count"].sum()))
col3.metric("Total Assets (M)", f"${plot_df['Total_assets_not_used'].sum()/1e6:.1f}M")

plot_df["Rank"] = plot_df["Score"].rank(method="min", ascending=False).astype(int)

display_columns = {
"Advisor_not_used": "Advisor Name",
"Advisor_IP_ID_not_used": "Advisor ID",
"Score": "Score",
"Client_count": "Client Count",
"Total_AUM": "Total AUM",
"External_assets": "External Assets",
"Rank": "Rank"
}

-----------------------------

Ranked Advisors Table

-----------------------------

st.subheader("Ranked Advisors")
st.dataframe(
plot_df.sort_values(by="Score", ascending=False)[list(display_columns.keys())]
.rename(columns=display_columns)
)

-----------------------------/////////////////////////////////////////////////////////////////////////////////////////////////////////////

Advisor Details

-----------------------------

st.subheader("Advisor Details")

Sort plot_df by Final_Score before populating the dropdown

sorted_plot_df = plot_df.sort_values(by="Score", ascending=False)

selected = st.selectbox("Select an advisor", sorted_plot_df["Advisor_not_used"])

if selected:
adv = plot_df[plot_df["Advisor_not_used"] == selected].iloc[0]
adv_scaled = rank_df[rank_df["Advisor_not_used"] == selected].iloc[0]

st.write(f"### {adv['Advisor_not_used']} - Final Score: {adv['Score']:.2f}")

import plotly.express as px

  
  
  Feature contributions


contributions = {}

for feature in norm_weights:

    if feature in adv_scaled:

        value = adv_scaled[feature]

        weight = norm_weights[feature]

        contribution = value * weight

        contributions[feature] = contribution

total_score = adv_scaled["Score"]

contribution_percentages = {

    k: (v / total_score) * 100 for k, v in contributions.items()

}

contrib_df = pd.DataFrame({

    "Feature": list(contribution_percentages.keys()),

    "Contribution (%)": [round(v, 1) for v in contribution_percentages.values()]

}).sort_values(by="Contribution (%)", ascending=False)

  
  
  Horizontal bar chart


bar_fig = px.bar(

    contrib_df,

    y="Feature",

    x="Contribution (%)",

    orientation='h',

    title=f"Feature Contribution to Score for {adv['Advisor_not_used']}",

    text=contrib_df["Contribution (%)"].apply(lambda x: f"{x}%"),

    labels={"Contribution (%)": "Contribution to Score (%)"}

)

st.plotly_chart(bar_fig, use_container_width=True)

  
  
  --- Pie Charts ---


col1, col2 = st.columns(2)

with col1:

    fig1, ax1 = plt.subplots()

    ax1.pie([adv["Total_AUM"], adv["External_assets"]],

            labels=["Our AUM", "External"],

            autopct="%1.1f%%", startangle=90, colors=["#4C72B0", "#55A868"])

    ax1.set_title("Assets Split")

    st.pyplot(fig1)

with col2:

    fig2, ax2 = plt.subplots()

    ax2.pie([adv["Clients_individual"], adv["Clients_team_not_used"]],

            labels=["Individual", "Team"],

            autopct="%1.1f%%", startangle=90, colors=["#FF6F61", "#6A5ACD"])

    ax2.set_title("Client Split")

    st.pyplot(fig2)

  
  
  --- Additional Metrics ---


st.write("Additional Metrics:")

st.write(f"- External Assets Share: {adv['Pct_external']:.2%}")

st.write(f"- Total AUM: ${adv['Total_AUM']:.0f}")

st.write(f"- Products per Client: {round(adv['Products_per_client'], 2)}")

st.write(f"- Rank: {adv['Rank']}")

--- Matplotlib Box Plot ---

st.subheader("Advisor Final Score Distribution (Box Plot)")
fig, ax = plt.subplots(figsize=(10, 4))
sns.boxplot(x=plot_df["Score"], ax=ax)
ax.set_title("Advisor Score Distribution")
ax.set_xlabel(" Score")
st.pyplot(fig)

--- Dynamic Percentage Inputs ---

st.subheader("Set Custom Class Thresholds (as Percentiles)")
pct_q1 = st.slider("Q1 upper bound (%)", min_value=0, max_value=100, value=25)
pct_q3 = st.slider("Q3 lower bound (%)", min_value=pct_q1, max_value=100, value=75)

--- Convert Percentiles to Score Thresholds ---

score_q1 = plot_df['Score'].quantile(pct_q1 / 100)
score_q3 = plot_df['Score'].quantile(pct_q3 / 100)

--- Classification Based on Thresholds ---

def classify_custom(score):
if score < score_q1:
return "Q1 (Lowest)"
elif score_q1 <= score <= score_q3:
return "Q2 (Middle)"
else:
return "Q3 (Highest)"

plot_df["Class"] = plot_df["Score"].apply(classify_custom)

--- Summary Stats by Class ---

summary = plot_df.groupby("Class").agg({
"Advisor_not_used": "count",
"Client_count": "sum",
"Total_AUM": "sum",
"Total_assets_not_used": "sum",
"External_assets": "mean",
"Products_per_client": "mean"
}).rename(columns={
"Advisor_not_used": "Advisor Count",
"Client_count": "Total Clients",
"Total_AUM": "Total AUM",
"Total_assets_not_used": "Total Assets",
"External_assets": "Avg External Assets",
"Products_per_client": "Avg Products per Client"
}).reset_index()

st.subheader("Class Summary")
st.dataframe(summary)

--- Advisor Lists by Class ---

for cls in ["Q1 (Lowest)", "Q2 (Middle)", "Q3 (Highest)"]:
st.markdown(f"### {cls} Advisors")
q_df = plot_df[plot_df["Class"] == cls]
st.write(q_df[[
"Advisor_not_used", "Score", "Client_count", "Total_AUM",
"Total_assets_not_used", "External_assets", "Products_per_client"
]].rename(columns={
"Advisor_not_used": "Advisor",
"Score": "Score",
"Total_assets_not_used": "Total Assets"
}))

--- Histogram of Recency ---

fig, ax = plt.subplots(figsize=(7, 4))
ax.hist(rank_df["Recency_days"], bins=10, color="#55A868", alpha=0.8, edgecolor='black')
ax.set_title("Distribution of Recency Across Advisors (Days)")
ax.set_xlabel("Days since last onboarding")
ax.set_ylabel("Number of Advisors")
st.pyplot(fig)

--- AUM-Based Filtering ---

st.subheader("Filter Advisors by AUM Range")

aum_min = st.number_input("Minimum AUM ($)", min_value=0, value=0, step=100000)
aum_max = st.number_input("Maximum AUM ($)", min_value=0, value=int(data["Total_AUM"].max()), step=100000)

apply_aum_filter = st.checkbox("Apply AUM Filter")

if apply_aum_filter:
plot_df = plot_df[
(plot_df["Total_AUM"] >= aum_min) &
(plot_df["Total_AUM"] <= aum_max)
]

pre_machine learning

YASHWANTH CHIKKI HD — Tue, 04 Nov 2025 09:18:02 +0000

Machine Learning Libraries - Quick Reference

Pandas

Reading Dataset

import pandas as pd

df = pd.read_csv("address.csv")
df = pd.read_excel("address.xlsx", sheet_name="sheet", usecols=['column1'])

Viewing Dataset

print(df.head())       # First 5 rows
print(df.tail())       # Last 5 rows
print(df.sample(5))    # Random 5 rows

Database Information

df.info()        # Gives full information
df.describe()    # Gives statistics
df.shape         # Returns (rows, columns)

Selecting Columns

column = df["column_name"]
subset = df[["column1", "column2"]]

Add Two Compatible Columns

df["new"] = df["col1"] + df["col2"]

Filter Rows

filtered = df[df['column'] > 10]

Merge Two DataFrames

newdf = pd.merge(df1, df2, on="common_column")

Missing Values

Check

values = df.isnull().sum()

Fill With Values

df['col'].fillna(0, inplace=True)

Drop Missing Values

df = df.dropna()

Duplicate Values

duplicates = df.duplicated().sum()

Add New Column

import numpy as np

y = np.array([1, 2, 3, 4])
df['col_new'] = y

Drop Column

df.drop(columns=['col_name'], inplace=True)

iloc Function

x = df.iloc[:, :-1]   # All but last column
y = df.iloc[:, -1]    # Only last column

NumPy Basics

import numpy as np

mean = np.mean(data)
st_dev = np.std(data)
median = np.median(data)
var = np.var(data)

Here’s a clean, professional README.md content formatted like a technical blog with proper Markdown structure — no pictorial symbols, only text and hierarchy.

Machine Learning Overview

1. Artificial Intelligence (AI)

Definition:
Artificial Intelligence is the branch of computer science that aims to create systems capable of performing tasks that typically require human intelligence.
These tasks include reasoning, problem-solving, perception, language understanding, and decision-making.

AI is the broader concept that encompasses Machine Learning (ML), Deep Learning (DL), and other intelligent decision-making techniques.

2. Machine Learning (ML)

Definition:
Machine Learning is a subset of Artificial Intelligence that enables systems to learn patterns and make decisions from data without being explicitly programmed.

Instead of using fixed rules, ML models improve their performance over time as they are exposed to more data.

3. Deep Learning (DL)

Definition:
Deep Learning is a specialized branch of Machine Learning that uses Artificial Neural Networks (ANNs) with multiple layers (hence “deep”) to learn complex patterns.
It is particularly effective for high-dimensional data such as images, speech, and natural language.

4. Difference Between AI, ML, and DL

Concept	Description	Example
AI	Broad concept of machines simulating intelligence	Voice Assistants like Alexa
ML	AI subset that learns from data	Spam email detection
DL	ML subset using neural networks	Image recognition in photos

5. Types of Machine Learning

Machine Learning can be categorized based on external supervision, training procedure, and type of learning.

5.1 Based on External Supervision

Tree Structure:

Machine Learning
│
├── Supervised Learning
│   ├── Regression
│   └── Classification
│
├── Unsupervised Learning
│   ├── Clustering
│   ├── Dimensionality Reduction
│   ├── Anomaly Detection
│   └── Association Rule Learning
│
├── Semi-Supervised Learning
│
└── Reinforcement Learning

Explanation:

Supervised Learning:
Models learn from labeled datasets, meaning each input has a known output.
Examples:
- Regression → Predicting continuous values (e.g., house price prediction)
- Classification → Predicting categorical labels (e.g., spam or not spam)
Unsupervised Learning:
Models learn from unlabeled data by finding hidden structures or patterns.
Examples:
- Clustering → Grouping similar data points (e.g., customer segmentation)
- Dimensionality Reduction → Reducing number of features (e.g., PCA)
- Anomaly Detection → Detecting outliers or rare events (e.g., fraud detection)
- Association Rule Learning → Discovering relationships (e.g., market basket analysis)
Semi-Supervised Learning:

A hybrid approach that combines a small amount of labeled data with a large amount of unlabeled data to improve learning accuracy.
Reinforcement Learning:

Learning through interaction with an environment, where the model receives rewards or penalties based on actions.

Common in robotics, gaming, and autonomous systems.

5.2 Based on Training Procedure

Tree Structure:

Machine Learning
│
├── Batch Learning (Offline Learning)
└── Online Learning

Explanation:

Batch Learning (Offline Learning):
The model is trained using the entire dataset at once.
Suitable for static data that doesn’t change over time.
Online Learning:
The model learns incrementally as data arrives in sequence.
Used in real-time systems or data streams (e.g., financial trading, live recommendation).

Key Concepts:

Learning Rate:
Controls how much model weights are adjusted after each iteration.
A small learning rate leads to slow convergence, while a large rate may cause instability.
Out-of-Core Learning:
Technique used when the dataset is too large to fit into memory.
The model processes data in small chunks sequentially.
River:
A modern Python library designed for online machine learning and data stream processing.
Vowpal Wabbit:
A highly efficient, open-source system for online learning, out-of-core learning, and reinforcement learning.

5.3 Based on Type of Learning

Tree Structure:

Machine Learning
│
├── Instance-Based Learning
└── Model-Based Learning

Explanation:

Instance-Based Learning:
The algorithm memorizes training examples and uses them directly to make predictions for new data.
Example: K-Nearest Neighbors (KNN)
Focuses on similarity measures (distance-based learning).
Model-Based Learning:
The algorithm builds an abstract model from training data and uses it for predictions.
Example: Linear Regression, Neural Networks
Focuses on generalizing patterns rather than memorizing examples.

6. Summary

Basis	Categories	Example Algorithms
External Supervision	Supervised, Unsupervised, Semi-Supervised, Reinforcement	Linear Regression, K-Means, Autoencoders, Q-Learning
Training Procedure	Batch, Online	SGD, River, Vowpal Wabbit
Type of Learning	Instance-Based, Model-Based	KNN, Decision Tree, Neural Network

7. Final Note

Machine Learning is a continuously evolving field that bridges mathematics, data, and computation.
Understanding the types, learning paradigms, and tools forms the foundation for building intelligent systems.

Data Analysis

Data Analysis is the process of inspecting, cleaning, transforming, and modeling data to uncover useful insights and support decision-making.
It helps identify trends, patterns, and relationships within data that drive business and research outcomes.

1. Statistics

Statistics forms the foundation of data analysis. It involves collecting, summarizing, interpreting, and presenting data in a meaningful way.

Key Concepts

Mean:
The average value of all data points.
Usage: Measures central tendency but is sensitive to outliers.
Median:
The middle value when data is sorted.
Usage: More robust to outliers; represents true center of skewed data.
Mode:
The most frequently occurring value.
Usage: Useful for categorical or discrete data.
Variance:
Measures how far data points are spread out from the mean.
Usage: Indicates the degree of data dispersion.
Standard Deviation (SD):
The square root of variance.
Usage: Shows how much data deviates from the mean in original units.

2. Univariate Analysis

Univariate analysis deals with the examination of one variable at a time to understand its distribution, central tendency, and spread.

2.1 Numerical Data

1. Histogram

Purpose: Shows the frequency distribution of numerical data.
What to See:
- Shape of distribution (normal, skewed, bimodal).
- Outliers or unusual gaps.
Next Steps:
- Apply normalization or log transformation if data is highly skewed.
- Handle outliers if they affect model performance.

2. KDE Plot (Kernel Density Estimate)

Purpose: Smoothed version of a histogram showing probability density.
What to See:
- Continuous estimation of data distribution.
- Peaks and spread of data.
Next Steps:
- Compare multiple KDEs to identify overlap or separation between features.
- Check if data follows a normal distribution.

3. Box Plot

Purpose: Displays five-number summary — minimum, Q1, median, Q3, and maximum — and detects outliers.
What to See:
- Outliers as points beyond whiskers.
- Interquartile range (spread of middle 50% of data).
Next Steps:
- Investigate or treat outliers.
- Compare distributions across different categories.

4. Q-Q Plot (Quantile-Quantile Plot)

Purpose: Checks whether data follows a specific theoretical distribution (commonly normal).
What to See:
- Points lying on a straight diagonal line → normal distribution.
- Deviations → skewness or heavy tails.
Next Steps:
- Apply transformation (log, square root) if data deviates strongly from normality.

2.2 Categorical Data

1. Bar Plot

Purpose: Displays frequency or proportion of categories.
What to See:
- Most/least frequent categories.
- Distribution imbalance.
Next Steps:
- Merge low-frequency categories if necessary.
- Encode categorical variables for modeling.

2. Pie Chart

Purpose: Represents percentage contribution of each category to the total.
What to See:
- Dominant categories.
- Proportional representation.
Next Steps:
- Verify if class imbalance exists.
- Consider resampling or reweighting for balanced learning.

3. Bivariate Analysis

Bivariate analysis explores the relationship between two variables — either both numerical, both categorical, or one of each.

3.1 Numerical vs Numerical

1. Scatter Plot

Purpose: Visualizes the correlation between two continuous variables.
What to See:
- Direction (positive, negative, none).
- Strength (tight clustering = strong relation).
- Outliers or nonlinear patterns.
Next Steps:
- Calculate correlation coefficient (Pearson/Spearman).
- Apply polynomial regression if relation is nonlinear.

2. Heatmap

Purpose: Shows correlation matrix using colors to indicate strength and direction of relationships.
What to See:
- High positive correlation close to +1.
- High negative correlation close to -1.
Next Steps:
- Remove highly correlated variables (multicollinearity).
- Select top correlated features for modeling.

3. Pair Plot

Purpose: Displays scatter plots for all numerical variable pairs with histograms on the diagonal.
What to See:
- Trends, clusters, and correlations.
- Distribution overlap between features.
Next Steps:
- Identify feature relationships for model input selection.
- Detect multicollinearity visually.

3.2 Categorical vs Categorical

1. Cross Tabulation (Contingency Table)

Purpose: Summarizes frequencies for combinations of two categorical variables.
What to See:
- Co-occurrence patterns between categories.
Next Steps:
- Perform Chi-Square test for independence.
- Identify significant associations.

2. Stacked Bar Chart

Purpose: Displays total count split by subcategories within each category.
What to See:
- Composition differences across groups.
Next Steps:
- Analyze class imbalance across subcategories.
- Visualize categorical interactions.

3. Grouped Bar Chart

Purpose: Compares multiple categorical groups side-by-side.
What to See:
- Relative differences between categories across groups.
Next Steps:
- Assess if certain combinations dominate.
- Use statistical tests to confirm relationships.

3.3 Numerical vs Categorical

Common Visualizations

Box Plot: Compare distribution of numerical values across categorical classes.
Violin Plot: Combines box plot and KDE for deeper distribution insight.
Bar Plot (with mean/median values): Shows aggregated numerical statistics by category.

What to See

Differences in spread and median across categories.
Overlapping distributions or distinct separation between classes.

Next Steps

If strong separation exists → good predictive potential.
If overlap is high → consider feature transformation or combining categories.

4. Summary

Analysis Type	Variable Type	Common Plots	Purpose
Univariate (Numerical)	Single numeric variable	Histogram, KDE, Box, Q-Q	Understand distribution and outliers
Univariate (Categorical)	Single categorical variable	Bar, Pie	Examine category frequency
Bivariate (Num vs Num)	Two numeric variables	Scatter, Heatmap, Pairplot	Explore correlations
Bivariate (Cat vs Cat)	Two categorical variables	Cross Tab, Stacked Bar, Grouped Bar	Check associations
Bivariate (Num vs Cat)	One numeric, one categorical	Box, Violin, Mean Bar	Compare group-wise distributions

Tensors

Definition (short): A tensor is a multi-dimensional array — a generalization of scalars, vectors, and matrices.

Rank (or ndims): number of dimensions (axes).
Shape: tuple listing length along each dimension.
Size (or number of elements): product of the shape elements.

0-D tensor (Scalar)

Rank: 0
Shape: ()
Size: 1
Example (NumPy):

import numpy as np
t0 = np.array(5)            # scalar
print(t0, "rank:", t0.ndim, "shape:", t0.shape, "size:", t0.size)
# Output: 5 rank: 0 shape: () size: 1

1-D tensor (Vector)

Rank: 1
Shape: (n,) where n is length
Size: n
Example (NumPy):

t1 = np.array([1.0, 2.0, 3.0])   # vector of length 3
print(t1, "rank:", t1.ndim, "shape:", t1.shape, "size:", t1.size)
# Output: [1. 2. 3.] rank: 1 shape: (3,) size: 3

2-D tensor (Matrix)

Rank: 2
Shape: (rows, cols)
Size: rows * cols
Example (NumPy):

t2 = np.array([[1, 2, 3],
               [4, 5, 6]])    # 2x3 matrix
print(t2, "rank:", t2.ndim, "shape:", t2.shape, "size:", t2.size)
# Output: [[1 2 3]
#          [4 5 6]] rank: 2 shape: (2, 3) size: 6

3-D tensor

Rank: 3
Shape: (d0, d1, d2) — e.g., (depth, rows, cols) or (samples, timesteps, features)
Size: d0 * d1 * d2
Example (NumPy):

t3 = np.array([
    [[ 1,  2,  3,  4],
     [ 5,  6,  7,  8],
     [ 9, 10, 11, 12]],
    [[13, 14, 15, 16],
     [17, 18, 19, 20],
     [21, 22, 23, 24]]
])  # shape (2, 3, 4)
print("rank:", t3.ndim, "shape:", t3.shape, "size:", t3.size)
# Output: rank: 3 shape: (2, 3, 4) size: 24

Notes on indexing and axes

Access: t2[row, col], t3[a, b, c].
Common axis semantics: axis=0 often denotes samples/batch, axis=1 features/time, etc. Always confirm library convention (NumPy, TensorFlow, PyTorch).

Challenges in ML Data Collection (short list)

APIs: rate limits, inconsistent formats, authentication, pagination. Mitigation: robust retry logic, caching, standardized parsers.
Web scraping: HTML structure changes, legal/robots rules, CAPTCHAs. Mitigation: use scraping frameworks, respect robots.txt, rotate user agents responsibly.
Insufficient data: small sample sizes that lead to overfitting. Mitigation: data augmentation, synthetic data, transfer learning.
Noisy / low-quality labels: human errors, label inconsistency. Mitigation: labeling guidelines, consensus labeling, label cleaning.
Class imbalance: skewed class distribution harming minority class performance. Mitigation: resampling, class weights, focal loss.
Missing values: gaps in the dataset. Mitigation: imputation, model choices robust to missingness, collect more data.
Data drift / distribution shift: training vs production mismatch. Mitigation: monitoring, periodic retraining, domain adaptation.
Privacy & compliance: PII and legal constraints. Mitigation: anonymization, differential privacy, lawful data sourcing.
Storage & scale: very large data requiring special infrastructure. Mitigation: out-of-core pipelines, cloud storage, streaming ingestion.

Machine Learning Development Lifecycle

Problem definition: clearly state the objective and success metrics.
Data collection: gather raw data from APIs, logs, sensors, scraping, databases.
Data ingestion & storage: reliably store and version raw data for reproducibility.
Data cleaning / preprocessing: handle missing values, noise, and format inconsistencies.
Exploratory Data Analysis (EDA): inspect distributions, relationships, and anomalies.
Feature engineering: transform, create, and select features that improve signal.
Model selection: choose algorithms suitable for the problem and data.
Training: fit models on training data with appropriate validation.
Validation & hyperparameter tuning: evaluate generalization and tune parameters.
Evaluation: measure performance on hold-out/test sets using defined metrics.
Deployment: package and serve the model into production environments.
Monitoring & maintenance: track performance, detect drift, and retrain as needed.
Documentation & governance: record data lineage, decisions, and compliance artifacts.

DATA STRUCTURES

YASHWANTH CHIKKI HD — Sun, 02 Nov 2025 12:55:17 +0000

python

YASHWANTH CHIKKI HD — Sat, 25 Oct 2025 19:29:43 +0000

Understanding Python Basics

Python is an interpreted, object-oriented, high-level,case-sensitive and strongly typed programming language.

Python code can either be:
- Directly interpreted and executed line by line, or
- Compiled into bytecode, which can then be executed on different machines.
Python file extension: .py

Variables in Python

A variable is a symbolic name assigned to a value — it acts as a reference (or pointer) to an object stored in memory.

a = 10
b = 10
print(id(a), id(b))

In Python, if two variables are assigned the same immutable value (like integers or strings), they both point to the same object in memory.
When you assign a new value to either variable, it stops pointing to the old object and instead points to a new one.
What’s Actually Happening? When you write:

a = 10
print(a)

Python does the following internally:

Creates an integer object
Assigns it to the variable a
Displays it in the console using the object’s class method (str)

To check the class/type of any object:

print(type(a))

Object Lifetime = An object exists as long as there is at least one variable referencing it
Once no variable points to it, Python’s garbage collector removes it automatically
Variable names cannot be Python keywords (like if, for, class, etc.)
Variable names must start with a letter or underscore (_), not a number
They are case-sensitive (Name and name are different)

Comments in Python

Comments are used to describe your code and make it more readable.

-Single-line comment
Starts with #

# This is a single-line comment

Multi-line comment You can use triple quotes (""") for multi-line comments:

"""
This is a
multi-line comment
"""

Data Types in Python

Python supports both primitive and non-primitive data types.

-Primitive Data Types

int → Integer numbers

a=10  #4bytes
b=int(10)
c=100l #8 bytes
#no type declaration needed

float → Decimal numbers

x=5.5 #4 bytes
x=float(5.5)

bool → Boolean (True / False)
str → String (text)

x="hello world"   #1 byte
x=str("hello world")
# Python has no char
# string is immutable, that is, every time you reassign a value new string is created

-complex

x=complex(1j)

Non-Primitive Data Types sequence type
list-> Ordered, mutable collection, non-homogeneous

x=["apple","orange"]
x=list(("apple","orange"))

tuple -> Ordered, immutable collection, non-homogeneous

x=("apple","orange")
x=tuple(("apple","orange"))
x=list/type((a,))
#faster than list

-array->ordered, mutable, homogenous

from array import array
my_array = array('i', [1, 2, 3, 4])

import numpy as np
arr = np.array([1, 2, 3, 4])

-range

x=range(6)
#creates a range object to be used in any function like 
z=list(x)
w=tuple(x)
for i in x:

mapping type

set-> Unordered, unique elements

x={"apple","banana"}
x=set(("apple","banana"))

dict-> Key-value pairs

x={"apple":"like","banana":"dislike"}
x=dict(("apple":"like","banana":"dislike"))

type conversion and type casting

type conversion
change in data type by itself
type casting
change in the datatype externally

random numbers

Python has no built-in random function, but

import random
print(random.randrange(1,10))

input function

-The input function always reads a string so we need to explicitly typecast it to an integer or a float

a=input("enter here")
print(type(a))
>><class 'str'>

a=input("enter")[4]
a=int(input("eneter a integer"))
print(a)
>><class 'int'>

math module

import math
math.sqrt(25)
math.pow(5)
math.ceil(3.3)
math.floor(3.5)
math.pi/e/etc

operators in Python

-arithmatic oparator

addition=a+b
substaraction=a-b
multiplication=a*b
division=a/b
modules=a%b -->gives reminder
exponential=a**b
floor division=a//b

a = 10
b = 3

print(a + b)   # 13
print(a - b)   # 7
print(a * b)   # 30
print(a / b)   # 3.333...
print(a // b)  # 3
print(a % b)   # 1
print(a ** b)  # 1000

assignment oparators
= -> x=5
+= -> x+=5 -> x=x+5
-= -> x-=5 -> x=x-5
= -> x=5 -> x=x*5
/= -> x/=5 -> x=x/5
//= -> x//=5 -> x=x//5
%= -> x%=5 -> x=x%5
= -> x=5 -> x=x**5

x = 10
x += 5   # 15
x -= 3   # 12
x *= 2   # 24
x /= 4   # 6.0
x //= 2  # 3.0
x %= 2   # 1.0
x **= 3  # 1.0 ** 3 = 1.0

-comparison operators

== Equal to → returns True if both values are equal
!= Not equal to → returns True if values are different
> Greater than
< Less than
>= Greater than or equal to
<= Less than or equal to

a = 10
b = 3

print(a == b)   # False
print(a != b)   # True
print(a > b)    # True
print(a < b)    # False
print(a >= b)   # True
print(a <= b)   # False

-logical operators
compares two or more conditions

and → Returns True if both conditions are true
or → Returns True if at least one condition is true
not → Reverses the result (True becomes False, False becomes True)

a = 10
b = 5

print(a > 3 and b < 10)   # True  (both conditions true)
print(a > 3 or b > 10)    # True  (one condition true)
print(not(a > 3))         # False (a > 3 is True, not makes it False)

-membership operators

in → Returns True if a value exists in a sequence (like list, string, tuple, etc.)
not in → Returns True if a value does not exist in a sequence

x = [1, 2, 3, 4, 5]

print(3 in x)        # True  (3 is present in the list)
print(10 in x)       # False (10 is not in the list)
print(10 not in x)   # True  (10 is not in the list)

-identity operators

is → Returns True if both variables refer to the same object in memory
is not → Returns True if variables refer to different objects in memory

a = [1, 2, 3]
b = a
c = [1, 2, 3]

print(a is b)       # True  (b points to the same list as a)
print(a is c)       # False (different objects with same content)
print(a is not c)   # True  (they are not the same object)

conditional statements

-if
-if,else
-if,elif,else

#  IF Statement

x = 10

# Normal way
if x > 5:
    print("x is greater than 5")

# One-liner way
if x > 5: print("x is greater than 5")


# IF–ELSE Statement

x = 3

# Normal way
if x % 2 == 0:
    print("Even")
else:
    print("Odd")

# One-liner way (ternary operator)
print("Even") if x % 2 == 0 else print("Odd")



# IF–ELIF–ELSE Chain

score = 85

# Normal way
if score >= 90:
    grade = "A"
elif score >= 75:
    grade = "B"
elif score >= 60:
    grade = "C"
else:
    grade = "D"
print(grade)

loops

-for loop
-while loop
-Python has no do-while, but it can be simulated


#for
for i in range(5):
    print(i)
#while
count = 0
while count < 5:
    print("Count:", count)
    count += 1

break
Purpose: Immediately stops the entire loop (for or while).
Effect: Control jumps outside the loop.
Use case: When you’ve found what you need and don’t want to continue.

for i in range(5):
    if i == 3:
        break
    print(i)
# Output: 0, 1, 2

continue
Purpose: Skips the current iteration and moves to the next one.
Effect: The loop doesn’t stop; it just jumps to the next cycle.
Use case: When you want to skip certain conditions but keep looping.

for i in range(5):
    if i == 2:
        continue
    print(i)
# Output: 0, 1, 3, 4

pass
Purpose: Does nothing — acts as a placeholder where code is syntactically required.
Effect: The loop or block runs but performs no action.
Use case: When you plan to write code later or need an empty block.

for i in range(5):
    if i == 2:
        pass  # nothing happens
    print(i)
# Output: 0, 1, 2, 3, 4

Functions

-simple functions

def function_name():
  print("idiot")
fucntion_name()

-function with arguments

def sumdata(a,b):
  print(a+b)
sumdata(1,5)

-functions with return

def sumdata(a,b):
  return(a+b)
print(sumdata(a,b))

oops

class human:
  leg = 2                                       #class variable
  def __init__(self,age,height):           #attributes/proprty                
    self.age=age                               #instance variable
    self.height=height
  def action(self):                         #methods/actions/behaviar
    print(self.age)
kishore=human(100,100)                       #1
kishore.action()/human.action(kishore)
Kishore.leg/human.leg
#1: We are calling a construction function called human, which is always the same as the class name

types of methods

class human:
  leg=2
  eyes=2
  def __init__(self,age,race,color):         #object initialization method
    self.race=race
    self.age = age
    self.color=color

  @class methods            #class methods(object not needed to call this) 
  def chnage_legs():
    self.legs=3

  @staticmethods
  def action(self):                         # normal methods
    print(self.race)

# Running the class method (no object needed)
Human.change_legs(3)
print(Human.legs)  # Output: 3

# Creating an object
Tarun = Human(22,"Indian", "Brown")

# Accessing instance attributes
print(Tarun.race)  # Output: Indian

# Running the static method
Human.action()  # or Tarun.action()