DEV Community: ALI MANSOOR

Your Database Is Slow Because Everything Is Hot

ALI MANSOOR — Sat, 16 May 2026 09:44:30 +0000

At some point, every growing system starts collecting ghosts.

So you have a job as a software engineer.

You build systems. APIs. Workers. Queues. Dashboards. Databases.

Life is good.

Well... not that good.

Somewhere in your infrastructure, there’s a database table quietly growing in size every second.

Maybe it's:

orders
logs
chat messages
notifications
webhook events
analytics

In the beginning, everything is beautiful.

Your queries are fast.
Your CPU is relaxed.
Your dashboards load instantly.

Then the data grows.

Good problem to have.

So naturally, you do what every engineer does.

You add indexes.

CREATE INDEX idx_orders_created_at
ON orders(created_at);

And suddenly:

Performance returns.
Life is good again.

For now.

Then Reality Arrives

A few months later:

New query patterns appear
More indexes get added
Old indexes become useless
Write performance starts dropping
Backups start dragging
CPU usage spikes
RAM usage starts looking offensive

And the database begins cursing at you in 0's and 1's.

The Internet Tells You To Shard Everything

So you go searching for answers.

Maybe you ask:

ChatGPT
Claude
Meta AI (I'm not judging)
that one senior engineer who says "just use Cassandra" (he's right tho)

And suddenly the suggestions begin:

partition the data
shard the database
horizontal scaling
replicas
sacrifice a goat to Kubernetes

ughhh.

maybe the issue is temperature...

Most Data Is Cold

This is the important realization.

A lot of production systems only actively use recent data.

Do you really need a 3-year-old webhook event sitting inside your primary production table?

Probably not.

But...

You still need to keep it.

For:

compliance
finance
legal
audits
analytics
"just in case"

It needs to go away (not actually go away, but still go away)

so what do you do?

Freeze The Data

The solution is surprisingly simple.

You stop treating old data like active data.

You freeze it.

Meaning:

keep recent data queryable
move old data elsewhere
reduce table size
reduce index size
reduce backup size
reduce IO pressure

Your database becomes smaller again.

Smaller databases are faster databases.

What Do You Mean Freeze It? Where Does Cold Data Go?

You have options.

Option 1 — Archive Tables

orders
orders_archive

Simple and effective.

Good when:

same database
rare access needed
low operational overhead

Option 2 — Data Warehouse

Perfect for:

BI teams
analytics
finance reporting

Examples:

BigQuery
Snowflake
ClickHouse
Redshift

Option 3 — Object Storage

Honestly?

Sometimes CSV files in S3 are enough.

Especially for:

logs
audit trails
compliance archives

You can export:

JSON
CSV

Cheap. Durable. Simple.

The Migration Strategy

So how do you do this?
If you dont know, I am worried about you....

You write a cron, it moves data which has surpassed the ttl to cold storage.

Do not move everything at once.

That is how you create incidents.

Instead:

small batches
gradual movement
continuous cleanup

In the beginning:

Run the cron every hour.

Move:

5k records
maybe 10k
maybe less

Observe:

lock times
replication lag
CPU spikes
IO usage

Then gradually increase retention movement.

Eventually:

daily jobs
weekly jobs
biweekly jobs

Your system stabilizes.

Referential Integrity Matters

Data is usually connected.

Example:

orders
├── payments
├── invoices
├── shipment_logs
└── audit_events

Moving only the parent record can create:

orphaned rows
broken analytics
failed joins
compliance problems

Archive related entities together.

I'll be back again (don't know when, to share some more insights...maybe another problem to solve)

Implementing K-Means Clustering from scratch in Python

ALI MANSOOR — Mon, 02 May 2022 10:21:23 +0000

Disclaimer! I am a student learning Datascience, Machine Learning. What I write here might have mistakes, do point them out in comments or reach out directly to me at my linkedin account.

What is K-Means Clustering?

k-means clustering is a method of vector quantization, originally from signal processing, that aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean (cluster centers or cluster centroid), serving as a prototype of the cluster. This results in a partitioning of the data space into Voronoi cells. - Wikipedia

If you did not understand this wikipedia definition like me, let me explain it in simpler terms.
In K-means clustering we divide n number of observations into k groups/clusters in such a way that the observations similar to each other are linked in one group.

Image Credits: Wikipedia

Steps for K-Means Clustering

Decide the value of k, which is the number of groups to divide your observations into.
Select k random points C (aka centroids) for each cluster within your observations.
Calculate absolute difference of each point from all centroids. |X-C|
Put the observation in the cluster which has the closest centroid.
Calculate new centroid for each cluster by taking average of all observations in that cluster.
Repeat step 3-5 until the centroids stop changing.
You have successfully organized n observations in k clusters.

I have also written the python code from scratch to implement k-means clustering for n-clusters, it currently works for 2-4 clusters(limited color values) but sometimes goes into infinite loop. if given n values of colors, it can work for n clusters

Github: https://github.com/TheAli711/datascience/tree/main/k-means-clustering

See you guys in the next article :)

Automatically deploy your react site with GitHub and Netlify

ALI MANSOOR — Sat, 02 Apr 2022 03:48:44 +0000

Hi guys, this is my first article on Dev.to platform
Today I am going to show you how you can automate your ReactJS deployments on Netlify.

Prerequisites:

NPM and NodeJS installed.
Understanding of React.
Understanding of git and GitHub.
Free Netlify and GitHub accounts.

Step 1: Setup ReactJS app

npx create-react-app my-portfolio

NPX

npx is a CLI tool whose purpose is to make it easy to install and manage dependencies hosted in the NPM registry. This is awesome because sometimes you just want to use some CLI tools but you don’t want to install them globally just to test them out. This means you can save some disk space and simply run them only when you need them.

Step 2: Set up GitHub repository

Login to your GitHub account and create a new repository, in the Initialize this repository with section leave everything unchecked

Step 3: Link your local code to GitHub repository

Go to the my-portfolio folder created earlier and do the following:
Rename local master branch to main:
git branch -M main
Add remote repository:
git remote add origin YourGithubRepoUrlHere
Push local code to GitHub:
git push -u origin main

Step 4: Link Netlify to GitHub repository

Login to your Netlify account.
Go to the Sites tab and click on Add New Site button.
Select Import existing project.
Select GitHub in Connect to Git Providers.
Pick the newly created repository.
Make sure that the branch to deploy is main
Click deploy site Your site will be deployed and now whenever there is any change main branch of GitHub, Netlify will automatically build and deploy your site with latest changes.

Bonus: Set up dev branch so you can create merge requests and get preview before deploying on the url

We can create a development branch in the repository where we can do all the changes and when we are confident with our code, we can create a merge request from development branch to main branch, before merging, Netlify will provide us with a deploy preview, on a separate URL, we can test it to see if our website functions as required and then can merge our code into main.

That's all for this article, Goodbye :)