DEV Community: Jeyashrisekar

Data in the Cloud: 6 Common Data Formats Every Data Analyst Should Know

Jeyashrisekar — Wed, 08 Oct 2025 13:53:59 +0000

In today’s data-driven world, information is stored, processed, and shared across multiple systems and cloud platforms. But have you ever wondered how this data is actually structured?

Different tools and systems prefer different data formats — each designed with a specific purpose. Whether you’re analyzing sales data in Python, querying with SQL, or running big data pipelines in Spark, understanding these formats will make your life much easier.

Let’s explore six of the most commonly used data formats in analytics:

CSV (Comma Separated Values)
SQL (Relational Table Format)
JSON (JavaScript Object Notation)
Parquet (Columnar Storage Format)
XML (Extensible Markup Language)
Avro (Row-based Storage Format)

Example Dataset
To make things concrete, let’s use a simple dataset representing students and their marks:

Name    Register_No Subject Marks
Alice   101 Math    88
Bob 102 Science 92
Charlie 103 English 79

We’ll represent this same dataset in six formats below.

1. CSV (Comma Separated Values)

What it is:
CSV is the simplest and most human-readable data format. Each line represents a record, and values are separated by commas.

Where it’s used:
Spreadsheets, Python data analysis (pandas), Excel, and cloud storage exports.

Example:

Name,Register_No,Subject,Marks
Alice,101,Math,88
Bob,102,Science,92
Charlie,103,English,79

*Pros:
*
Easy to read and create
Supported by almost every tool

Cons:
No data types (everything is text)
Doesn’t handle nested data well

2. SQL (Relational Table Format)

What it is:
SQL is used to store data in relational databases (like MySQL, PostgreSQL, or SQLite). Data is organized into tables with rows and columns.

Where it’s used:
Database systems, analytics platforms, and data warehouses.

Example:

CREATE TABLE Students (
  Name VARCHAR(50),
  Register_No INT,
  Subject VARCHAR(50),
  Marks INT
);

INSERT INTO Students (Name, Register_No, Subject, Marks) VALUES
('Alice', 101, 'Math', 88),
('Bob', 102, 'Science', 92),
('Charlie', 103, 'English', 79);

Pros:
Strong schema and structure
Easy querying with SQL language

Cons:
Not flexible for nested or complex data

3. JSON (JavaScript Object Notation)

What it is:
JSON is a lightweight format for structured data. It represents objects as key-value pairs and is widely used in web APIs and NoSQL databases.

Where it’s used:
APIs, web apps, NoSQL systems (MongoDB, Firebase).

Example:

[
  { "Name": "Alice", "Register_No": 101, "Subject": "Math", "Marks": 88 },
  { "Name": "Bob", "Register_No": 102, "Subject": "Science", "Marks": 92 },
  { "Name": "Charlie", "Register_No": 103, "Subject": "English", "Marks": 79 }
]

Pros:
Flexible and supports nested data
Readable and language-independent

Cons:
Larger file size than CSV
Slower for big data analytics

4. Parquet (Columnar Storage Format)

What it is:
Parquet is a binary, columnar data format optimized for analytical workloads. Instead of storing row by row, it stores data column by column — making queries on specific fields lightning-fast.

Where it’s used:
Big data systems like Apache Spark, Hive, AWS Athena, and Google BigQuery.

Example (conceptual view):

# Parquet is binary, but conceptually looks like this:

Column: Name        → [Alice, Bob, Charlie]
Column: Register_No → [101, 102, 103]
Column: Subject     → [Math, Science, English]
Column: Marks       → [88, 92, 79]

Pros:
Highly compressed and efficient
Ideal for large datasets and analytics

Cons:
Not human-readable
Harder to debug manually

5. XML (Extensible Markup Language)

What it is:
XML uses tags to define structured data, similar to HTML. It’s widely used in configuration files, legacy systems, and data interchange.

Where it’s used:
Web services (SOAP), enterprise applications, configuration files.

Example:

<Students>
  <Student>
    <Name>Alice</Name>
    <Register_No>101</Register_No>
    <Subject>Math</Subject>
    <Marks>88</Marks>
  </Student>
  <Student>
    <Name>Bob</Name>
    <Register_No>102</Register_No>
    <Subject>Science</Subject>
    <Marks>92</Marks>
  </Student>
  <Student>
    <Name>Charlie</Name>
    <Register_No>103</Register_No>
    <Subject>English</Subject>
    <Marks>79</Marks>
  </Student>
</Students>

Pros:
Structured and self-descriptive
Supports nested and hierarchical data

Cons:
Verbose and larger file sizes
Harder to parse compared to JSON

6. Avro (Row-based Storage Format)

What it is:
Avro is a row-based binary format developed under Apache Hadoop. It stores both the data and schema together — great for streaming and serialization.

Where it’s used:
Apache Kafka, Hadoop, and data pipelines.

Example (conceptual view):

{
  "type": "record",
  "name": "Student",
  "fields": [
    {"name": "Name", "type": "string"},
    {"name": "Register_No", "type": "int"},
    {"name": "Subject", "type": "string"},
    {"name": "Marks", "type": "int"}
  ]
}

# Data:
[
  {"Name": "Alice", "Register_No": 101, "Subject": "Math", "Marks": 88},
  {"Name": "Bob", "Register_No": 102, "Subject": "Science", "Marks": 92},
  {"Name": "Charlie", "Register_No": 103, "Subject": "English", "Marks": 79}
]

Pros:
Schema evolution support
Compact and fast for streaming

Cons:
Binary (not human-readable)
Needs libraries to read/write

Final Thoughts

Each of these formats serves a specific purpose.
Use CSV when you need simplicity.
Choose JSON for flexibility.
Rely on SQL for structured, relational data.
Go with Parquet or Avro for big data or cloud-based analytics.
And if you’re maintaining older systems, XML might still be your go-to.
Understanding these formats helps you move data seamlessly across systems — a crucial skill in modern cloud-based analytics.

Exploring MongoDB - Great Start to learn Data Analytics

Jeyashrisekar — Mon, 25 Aug 2025 14:42:48 +0000

Hi there,

As a beginner in databases, I recently started exploring MongoDB, a popular NoSQL database. Unlike traditional SQL databases, MongoDB stores data in flexible JSON-like documents, making it a great choice for modern applications that need scalability and flexibility.

1. STARTING
I started my MongoDB journey by installing the MongoDB. Then I created a Database and Collection named Business and review respectively.

2. IMPORTING THE DATASET
Then I imported my "yelp.json" which have 1000 rows.

3. INSERTING THE MANY DATASETS through insertMany()
After importing the dataset, I had inserted 10 dataset using insertMany().

After insertion, i got the output like this

3. Query to find top 5 businesses with highest average rating

Then I found Top 5 businesses with highest average rating after perform aggregate function

db.review.aggregate([ { $group: { _id: "$business_id", avg_rating: { $avg: "$stars" } } }, { $sort: { avg_rating: -1 } }, { $limit: 5 } ])

5. Query to count how many reviews contain the word “good”.

I used countDocuments to cound the reviews contain "good"

db.review.countDocuments({ text: /good/i })

6. Query to get all reviews for a specific business ID.

I get all reviews for a specific business ID using find()

db.review.find({ business_id: "01" })

7. Update a review and delete a record
7.1: Update 1 dataset using updateOne()

db.review.updateOne( { review_id: "A02" }, { $set: { stars: 5, text: "Updated: Excellent food and very friendly staff." } } )

7.2: Delete 1 dataset using deleteOne()

db.review.deleteOne({ review_id: "E01" })

Conclusion

This exercise gave me a clear understanding of how MongoDB handles real-world data in a flexible way. From inserting and retrieving reviews to performing text searches, aggregations, updates, and deletions — MongoDB makes working with data simple and powerful.

For Data Engineers and Developers, MongoDB is an essential tool to master, as it allows storing large-scale, unstructured, or semi-structured data efficiently. This small project was just the beginning of my #LearningJourney, and I look forward to exploring advanced MongoDB features like indexing, schema validation, and aggregation pipelines in the future.

JENKINS

Jeyashrisekar — Thu, 07 Nov 2024 21:00:37 +0000

Overview:

Jenkins is a tool that is used for automation. It is mainly an open-source server that allows all the developers to build, test and deploy software. It is written in Java and runs on java only. By using Jenkins we can make a continuous integration of projects(jobs) or end-to-endpoint automation.
Jenkins is an open source continuous integration tool written in Java. Jenkins provides continuous integration services for software development. It is a server-based system running in a servlet container such as Apache Tomcat.

Key Features

Jenkins offers many attractive features for developers:

Easy Installation
Jenkins is a platform-agnostic, self-contained Java-based program, ready to run with packages for Windows, Mac OS, and Unix-like operating systems.
Easy Configuration
Jenkins is easily set up and configured using its web interface, featuring error checks and a built-in help function.
Available Plugins
There are hundreds of plugins available in the Update Center, integrating with every tool in the CI and CD toolchain.
Extensible
Jenkins can be extended by means of its plugin architecture, providing nearly endless possibilities for what it can do.
Easy Distribution
Jenkins can easily distribute work across multiple machines for faster builds, tests, and deployments across multiple platforms.
Free Open Source
Jenkins is an open-source resource backed by heavy community support.

How it fits into Devops/DevSecOps

Jenkins is critical to DevOps due to its CI/CD capabilities. Continuous integration is a software development practice developers use to incorporate code changes into a central repository regularly. Continuous delivery (CD) helps deliver reliable applications in short cycles anytime. With the help of the Jenkins pipeline, DevOps teams can accelerate continuous software delivery by automating simple deployment processes.
With Jenkins in DevOps, teams can automate testing codes by running scripts and can integrate them with various test automation tools. As highlighted Jenkins supports a wide range of testing frameworks and tools, and DevOps teams can thus choose ones that best fit their needs. Jenkins can be configured to run tests automatically in case of code changes.
Jenkins in DevOps helps test any identified issues simultaneously even in the case of massive deployments. Jenkins DevOps model automates this process as every piece of merged code is mainly at a production-ready stage at any point.

Resources are empowered as it gives them complete workflow control and effectively managing the CI/CD pipeline. Together with enhanced collaboration, shortening the SDLC is one of the primary objectives of DevOps. Let us remember that manual testing is slow and is prone to cause bottlenecks in DevOps, calling for the need to automate the testing process. The integration of codeless test automation with DevOps and Agile approaches can be used to create automated tests that can run continuously, providing quick feedback on the quality of the application.

Programming Languages

The progarmming language is used got Jenkin is Java programming

Parent Company of the Jenkins tool

**
The CD Foundation, an organization within the Linux Foundation, governs the development of Jenkins as an open source project.
Jenkins was originally developed by Kohsuke Kawaguchi, a Java developer at Sun Microsystems, under the name Hudson. After Oracle acquired Sun, Kawaguchi left the company and continued the project under the name Jenkins because Oracle owned the rights to Hudson.

Is Jenkins Open source or paid one

Jenkins is a open source

DEV Community: Jeyashrisekar

Data in the Cloud: 6 Common Data Formats Every Data Analyst Should Know

1. CSV (Comma Separated Values)

2. SQL (Relational Table Format)

3. JSON (JavaScript Object Notation)

4. Parquet (Columnar Storage Format)

5. XML (Extensible Markup Language)

6. Avro (Row-based Storage Format)

Final Thoughts

Exploring MongoDB - Great Start to learn Data Analytics

Conclusion

JENKINS

Overview:

Key Features

How it fits into Devops/DevSecOps

Programming Languages

Parent Company of the Jenkins tool

Is Jenkins Open source or paid one

Jenkins Logo