Deepening My Roots in the Data Ecosystem - Choosing Depth Over Breadth

The Original Plan vs. Reality
In my previous post, I planned to step into LLM orchestration by contributing to LangChain or diving into full-stack development with Django. However, digging into the codebase made me realize a distinct difference in engineering styles.

The library relies heavily on abstraction layers to wrap various LLMs. While this is architecturally impressive, I found that I didn't get the same satisfaction as I did when working with scikit-learn or Dagster. It wasn't just about complexity, it was about the nature of the code. I realized that I prefer the logic of data pipelines and algorithms over the integration-heavy nature of LLM wrappers.

Rediscovering the Joy of Data Engineering
Naturally, I shifted my focus back to Dagster. Scanning the issue tab, I found myself drawn to problems that dealt with strict data flow and orchestration logic.

It wasn't just because Dagster was familiar, it was because the challenges were genuinely more stimulating. For instance, working on a feature that required learning Polars was exciting, even though it was a completely new library for me. This confirmed my preference:

"I enjoy the process when working on the concrete logic of data processing rather than the abstraction layers of LLM applications."

Choosing Depth Over Breadth
I made a strategic decision. Instead of making surface-level contributions in a new repository, I decided to double down on Dagster. This allowed me to move beyond minor patches and focus on high-impact work.
I focused on:

1. Resolving Core Issues: Diving deep into the internal logic to fix bugs that were blocking other users.
2. Expanding Functionality: Implementing new features that enhance the tool's usability.

Leveraging my previous experience with the codebase allowed me to use the time more efficiently. I could navigate the source code with intuition, enabling me to tackle complex architectural problems that would have been out of my reach just a few months ago.

Finding My Path
This journey took an unexpected turn, but it taught me a valuable lesson. Being a skilled developer isn't about following the latest trends; it's about recognizing your strengths and doubling down on them. Instead of spreading myself thin, I chose to deepen my expertise in the data ecosystem.