Why do I learn Apache Spark as I move from Data Analyst to Data Engineer?

Ashwin Udhayakannan — Wed, 08 Jul 2026 15:47:15 +0000

Hey guys... I'm AshwinUdhayakannan, a guy trying to upskill in the world of information technology. I'm a Data Analyst with 2 years of experience working with Blockchain data. I'm planning to switch into data engineering, so instead of just guessing what to learn next, I scoured job postings, roadmaps, and articles to see what tools and technologies were actually in demand.

A bunch of names kept showing up — Airflow, dbt, Kafka — but one tool kept appearing almost everywhere I looked: Apache Spark. Whether it was job descriptions asking for it directly, or engineers online calling it a "must-know" for the role, Spark kept winning the popularity contest by a wide margin.

Naturally, that made me curious. When I started reading about it, my first thought was "okay, this is basically just a faster version of Hadoop." Turns out that's only half true — Spark processes data in-memory instead of constantly writing to disk, which is a big part of why it's so much faster, but there's clearly more to it than just speed. That's exactly the kind of gap I want to close.

I also thought about why Spark specifically, over the other tools I found. Airflow and dbt are more about orchestration and transformation logic — they didn't feel like the foundational skill I needed first. Spark, on the other hand, kept coming up as the engine actually doing the heavy lifting behind a lot of pipelines, which felt like the right place to start given where I want to head — data engineering now, and possibly AI/ML down the line.

From what I've gathered so far, Spark shows up everywhere — ETL/ELT pipelines, real-time streaming, big data analytics, even machine learning pipelines. Companies like Netflix and Uber have written publicly about using it at scale, which tells me it's not just a resume keyword — it's genuinely load-bearing infrastructure in the industry.

So here's what I'm doing: learning Apache Spark from scratch and documenting it as I go, as a series of blogs. Not polished explainers — just my honest learnings, confusions, and "Oops" moments along the way.

Next up: "Apache Spark 101: What It Is, Where It's Used, and Why Everyone's Talking About It", in my own words, as I understand it right now.

If you're on a similar journey or already know Spark well, I'd love to hear your thoughts in the comments.

DEV Community: Ashwin Udhayakannan

Why do I learn Apache Spark as I move from Data Analyst to Data Engineer?