Hello dev.to! I'm a Software Engineer - 1, working on a multi-tenant e-commerce data platform — building and maintaining pipelines that pull data from source systems, transform it, and load it into Snowflake, all orchestrated through Airflow.
I'm starting this blog to write about the real, specific problems I run into day to day - not just "what is Airflow" explainers, but the actual trade-offs and debugging stories that come up when you're running pipelines in production for multiple tenants at once.
What I actually work on
My day-to-day sits across a few areas:
- Orchestration - designing and maintaining Airflow DAGs, including containerized task execution with DockerOperator
- Data ingestion - pulling data from external source systems into a warehouse, and dealing with everything that goes wrong along the way (schema drift, timezone mismatches, duplicate records, silent failures)
- SQL & Snowflake - flattening semi-structured data, modeling header/detail relationships, writing queries that hold up as data volume grows
- Python - scripts and utilities that keep the pipeline reliable: preflight checks, validation, cleanup jobs
What you can expect from this blog
I'll be writing short, practical posts based on real problems I've hit - things like:
- Choosing between different Airflow operators for different kinds of tasks
- Parallelizing pipelines without causing resource collisions
- Debugging "the data looks wrong" tickets systematically
- SQL patterns for messy, semi-structured source data
No fluff, no "10x your pipeline" clickbait - just what actually worked (and what didn't) when I tried it.
Why I'm writing this
Two reasons: first, writing things down is one of the best ways to actually understand them properly - explaining a concept clearly forces me to know it clearly. Second, I want to build a small, honest record of what work actually looks like day to day, for anyone earlier in their career trying to figure out what this field involves.
If you're into ETL pipelines, or getting into the field, or just curious how ETL pipelines actually break and get fixed - stick around. First technical post coming soon: a practical comparison of Airflow's DockerOperator and PythonOperator, and when to reach for each.
Feel free to say hi in the comments - always happy to connect.
Top comments (0)