DEV Community: Sujitha Selvaraj

🧹 Data Cleaning Challenge with Pandas (Google Colab)

Sujitha Selvaraj — Sun, 09 Nov 2025 13:03:47 +0000

🧹 Data Cleaning Challenge with Pandas (Google Colab)
Data cleaning is one of the most crucial steps in any data science or analytics project. In this challenge, I worked on a real-world dataset from Kaggle with over 100,000 rows, performing various Pandas operations to clean, preprocess, and prepare it for further analysis.

📂 Dataset Details
For this challenge, I selected the E-commerce Sales Dataset from Kaggle containing around 120,000 rows and 12 columns.

It includes data such as:

🧾 Order ID
👤 Customer Name
🛒 Product & Quantity
💰 Sales & Discount
🌍 Region
📅 Order Date

Before Cleaning:

Rows → 120,000
Columns → 12
File format → .csv

⚙️ Tools & Environment
Python 3
Google Colab
Libraries: Pandas, NumPy, Matplotlib

Exploring NoSQL Data Analysis: A Practical Study Using a Kaggle Dataset

Sujitha Selvaraj — Sun, 09 Nov 2025 13:00:00 +0000

🗂️ Step 1: Setting up MongoDB Atlas
Go to MongoDB Atlas.
Create a free cluster (use the Shared Tier option).
Under Network Access, add your IP:
Click Network Access → Add IP Address → Allow access from anywhere (0.0.0.0/0).
Create a database user and remember the credentials. Example:

Username: 22cs181
Password: Sujitha

Once your cluster is ready, click “Connect → Connect using MongoDB Shell” and copy the connection string.

💻** Step 2: Connect from Mongo Shell**
Open PowerShell or Command Prompt, then run:

`bash
mongosh "mongodb+srv://m0.wpjmxqh.mongodb.net/" --apiVersion 1 --username 22cs181_db_user

Then enter your password when prompted:

Enter password: Sujitha

If connection succeeds, you’ll see:

Atlas atlas-xxxx-shard-0 [primary]>

📥 Step 3: Create a Database and Insert Records
Switch to a database (it will auto-create):

javascript
use businessDB

Insert 10 sample business review records:

javascript
db.reviews.insertMany([
{ "business_id": "B001", "name": "Cafe Aroma", "rating": 4.6, "review": "Good food and fast service!", "date": "2025-11-07" },
{ "business_id": "B002", "name": "Pizza Palace", "rating": 4.8, "review": "Amazing crust and cheese quality!", "date": "2025-11-07" },
{ "business_id": "B003", "name": "Tea Time", "rating": 4.2, "review": "Nice ambience and friendly staff.", "date": "2025-11-07" },
{ "business_id": "B004", "name": "Sweet Treats", "rating": 3.9, "review": "Desserts were good but service was slow.", "date": "2025-11-07" },
{ "business_id": "B005", "name": "Veggie Delight", "rating": 4.1, "review": "Healthy food with good taste.", "date": "2025-11-07" },
{ "business_id": "B006", "name": "Burger Hub", "rating": 4.9, "review": "Best burgers ever!", "date": "2025-11-07" },
{ "business_id": "B007", "name": "Ocean Dine", "rating": 4.7, "review": "Fresh seafood and great view.", "date": "2025-11-07" },
{ "business_id": "B008", "name": "Spice Route", "rating": 3.8, "review": "Food was okay, but spicy.", "date": "2025-11-07" },
{ "business_id": "B009", "name": "Bakers Street", "rating": 4.5, "review": "Good pastries and coffee.", "date": "2025-11-07" },
{ "business_id": "B010", "name": "Quick Bite", "rating": 4.0, "review": "Good service and clean place.", "date": "2025-11-07" }
])

🔍 Step 4: Queries
🏆 4.1 Top 5 Businesses by Rating
javascript db.reviews.find().sort({ rating: -1 }).limit(5)

🔤 4.2 Count of Reviews Containing “good”
javascript db.reviews.countDocuments({ review: /good/i })

🏪 4.3 Get Reviews for a Specific Business ID
javascript db.reviews.find({ business_id: "B005" })

✏️ Step 5: Update and Delete
✏️ Update a Review
javascript db.reviews.updateOne( { business_id: "B005" }, { $set: { rating: 4.3, review: "Updated: Great taste and fresh ingredients!" } } )

🗑️ Delete a Record
javascript db.reviews.deleteOne({ business_id: "B010" })

📤** Step 6: Export Data to JSON/CSV**
Exit Mongo shell:

bash exit

Then run the following from PowerShell (not inside mongosh) 👇

📄 Export as CSV
bash mongoexport --uri="mongodb+srv://22cs098_db_user:NAVEEN@m0.wpjmxqh.mongodb.net/businessDB" --collection=reviews --type=csv --fields=business_id,name,rating,review,date --out=reviews.csv

📦 Export as JSON
bash mongoexport --uri="mongodb+srv://22cs098_db_user:NAVEEN@m0.wpjmxqh.mongodb.net/businessDB" --collection=reviews --out=reviews.json

📊** Step 7: View the Exported Files**
Open reviews.csv in Excel or VS Code.
Open reviews.json in any text editor.

🎯** Final Thoughts**
MongoDB Atlas makes it easy to:

Manage cloud-hosted databases
Perform CRUD operations
Export results in multiple formats
This project demonstrates all essential MongoDB operations — perfect for Data Engineering and Database Management learning tasks.

Data in the Cloud — 6 Common Data Formats Every Analyst Should Know

Sujitha Selvaraj — Thu, 06 Nov 2025 12:54:14 +0000

1. CSV (Comma Separated Values)

What it is:
CSV is the simplest and most widely used data format. It stores data in plain text where each line represents a record, and values are separated by commas.

Example:

name,reg_no,subject,marks
Asha Rao,R001,Maths,89
Vikram S,R002,Physics,76
Meera K,R003,Chemistry,92
Rohit P,R004,Maths,68

Where it’s used:
CSV is used in spreadsheets, data imports/exports, and small-scale analytics.

2. SQL (Relational Table Format)

What it is:
SQL represents data stored in relational databases. The data is organized in tables with defined columns and data types. Each row represents one record.

Example:

TABLE: students

name | reg_no | subject | marks

Where it’s used:
Used in databases like MySQL, PostgreSQL, and SQL Server for structured data and transactional operations.

3. JSON (JavaScript Object Notation)

What it is:
JSON stores data as key-value pairs. It is lightweight, human-readable, and commonly used in APIs and modern web applications.

Example:

[
{"name": "Asha Rao", "reg_no": "R001", "subject": "Maths", "marks": 89},
{"name": "Vikram S", "reg_no": "R002", "subject": "Physics", "marks": 76},
{"name": "Meera K", "reg_no": "R003", "subject": "Chemistry", "marks": 92},
{"name": "Rohit P", "reg_no": "R004", "subject": "Maths", "marks": 68}
]

Where it’s used:
APIs, web applications, configuration files, and NoSQL databases like MongoDB.

4. Parquet (Columnar Storage Format)

What it is:
Parquet is a columnar storage format used for big data analytics. Unlike row-based formats, Parquet stores data column-wise, which reduces storage space and increases query performance.

Example (conceptual view):

name reg_no subject marks
Asha Rao R001 Maths 89
Vikram S R002 Physics 76
Meera K R003 Chemistry 92
Rohit P R004 Maths 68

Where it’s used:
Big data platforms like Apache Spark, Hadoop, and AWS Athena for fast analytics and cloud storage efficiency.

5. XML (Extensible Markup Language)

What it is:
XML is a tag-based format used to represent structured data. It is similar to HTML but designed to store and transport data rather than display it.

Example:

Asha Rao
R001
Maths
89

Vikram S
R002
Physics
76

Where it’s used:
Web services (SOAP), configuration files, and systems that require strong data validation through schemas.

6. Avro (Row-based Storage Format)

What it is:
Avro is a compact binary format that stores both data and schema. It’s designed for fast data serialization and supports schema evolution, making it ideal for real-time data pipelines.

Example (logical representation):

{"name": "Asha Rao", "reg_no": "R001", "subject": "Maths", "marks": 89}
{"name": "Vikram S", "reg_no": "R002", "subject": "Physics", "marks": 76}
{"name": "Meera K", "reg_no": "R003", "subject": "Chemistry", "marks": 92}
{"name": "Rohit P", "reg_no": "R004", "subject": "Maths", "marks": 68}

Where it’s used:
Data streaming (Apache Kafka), data serialization, and large-scale data pipelines.

Conclusion:

Each data format serves a different purpose in the analytics and cloud ecosystem:

CSV is simple and universal.
SQL ensures structure and relationships.
JSON adds flexibility and nesting.
Parquet optimizes analytical queries.
XML emphasizes structure and validation.
Avro focuses on efficient, schema-based data transport.

Understanding when and how to use these formats is a core skill for any data analyst, data engineer, or cloud professional.