DEV Community: Leo Chashnikov

Boosting Career in Data Engineering: Insights and Strategies

Leo Chashnikov — Mon, 13 Nov 2023 13:08:46 +0000

My name is Leonid and I have been working as a developer for over 10 years, currently at Meta (ex-Facebook). The term "Data Engineering" best describes the scope of my responsibilities at the moment.

In this article, I’ll share tips on building a career as a Data Engineer and delve into the dos and don'ts.

How is a Data Engineer (DE) different from a Software Engineer-a (SWE) and a Data Scientist-a (DS)?

The typical role of a Data Engineer involves constructing a data processing pipeline and optimizing it. This encompasses designing an effective data storage schema, ensuring seamless data updates. In certain companies, Data Engineers also develop systems and frameworks to facilitate Data Scientists in deploying their models and experimenting with data more effortlessly.

Distinguishing itself from Software Engineering, Data Engineering hinges on a profound grasp of distributed systems, data formats, and data processing procedures. It's common to find individuals transitioning from the Software Engineering realm to Data Engineering, as the languages may be the same but with different frameworks.

On the other hand, Data Scientists possess a profound understanding of the domain and business, crucial for effective data processing and extracting insights. While they may have programming skills, it's not the primary focus. Data Scientists often use "notebooks" and scripts for testing theories, sometimes falling short of engineering standards and optimal code.

In a collaborative setup, Data Engineers work closely with Data Scientists in the same team. Their responsibilities encompass deploying and potentially rewriting code, ensuring stability and speed, and addressing all potential corner cases.

Who and why might find the transitioning to Data Engineering appealing?

**
Delving into Data Engineering can be particularly thrilling for developers with a keen eye for detail and a passion for optimization. It offers an opportunity to explore the intricacies of distributed systems and efficient data processing, allowing Software Engineers to enhance their skills with SQL and NoSQL databases.

Additionally, for those eager to collaborate closely with Data Scientists, transitioning to Data Engineering opens doors to jointly refine and optimize analytical models. It requires the ability to tackle tasks related to code efficiency, stability, and accounting for all conceivable corner cases.

What adds an intriguing dimension is the connection with the business aspects of a company. Data Engineers must comprehend business needs and domain areas to ensure effective data processing aligns with the company's strategy, delivering tangible value. This holistic approach makes Data Engineering appealing to those seeking to blend technical prowess with a strategic inclination to influence business processes through data processing optimization.

How to transition from a Software Engineer to a Data Engineer?

If you're making the shift, your familiarity with tools commonly used by Data Engineers is a valuable asset. Having experience with diverse databases proves beneficial, particularly in discerning the most suitable database for specific scenarios. For instance, PostgreSQL might excel for feature-rich searches, while ElasticSearch could be more efficient for text searches. A solid grasp of SQL is essential, as SQL-like queries are standard across various databases and data processing systems like Spark or query systems such as Presto.

Embracing standard engineering practices is fundamental; this includes writing tests—ranging from unit tests to integration tests—adopting CI/CD practices, and implementing infrastructure-as-code. This distinction sets Data Engineers apart from Data Scientists, who often focus on one-time code creation with less emphasis on long-term support.

In the realm of data tools, pinpointing a single dominant stack is challenging, but a good starting point may include:

Apache Spark: A highly popular platform for processing data in both batch and streaming modes.
Apache Kafka: A distributed event streaming platform that also allows on-the-fly data processing, serving as an alternative to Spark in a streaming system.
Presto or Snowflake: Query and access systems for data. Snowflake stores data itself, while Presto facilitates connections to different databases and the combination of their data.
Apache Airflow: A widely used platform for managing dependencies between diverse data sources and pipelines for processing them.

What’s expected at higher levels?

1. Understanding of Business and Domain Needs
Communicating effectively with the primary clients, typically data scientists or data analysts, requires a grasp of the business aspect of your company. Understanding the company's operations and revenue sources enhances the ability to discern critical data and its intended use.

2. Greater Planning Horizon
Elevating to a senior level involves not just coding proficiency (expected at the middle level) but the capability to break down large projects, envision the broader picture, and navigate tradeoffs. Senior roles often necessitate acting as an "arbitrator" in resolving technical disputes within the team.

3. Risk Management
As projects grow in size, unforeseen issues and on-the-fly changes to technical tasks become more prevalent. Effective risk management becomes crucial. This might involve rapid Proof of Concepts (PoCs) to test the viability of ideas and adapt the original design as needed.

4. Mentoring and Knowledge Sharing
Despite individual proficiency, the ability to accomplish the work of 10 people solo is limited. Senior developers should evolve into mentors, sharing knowledge and assigning tasks to individuals who can derive the most benefit. An indispensable senior who hoards knowledge can hinder the team's growth, depriving less-experienced members of engaging tasks.

How to advance further in your career?

1. Define Your Growth Path
Identify where you want to grow and why. Some may prefer the path of an Individual Contributor, deeply engaged in coding and project planning. Others may lean towards managerial roles, focusing on removing obstacles for developers and fostering collaboration with other teams. It's essential to understand your preference and motivation.

2. Find a mentor
Find someone a few steps ahead on the path you've chosen. Having a mentor, especially within the same company, is valuable for insights into the promotion process, understanding what is highly valued, and presenting yourself effectively.

3. Explore Beyond Your Work Environment
Working in a stable company often limits exposure to a specific technical stack. Seek opportunities outside of work to try new things—whether through open source projects or personal endeavors. Experimenting with new technologies, even in small projects, allows you to familiarize yourself with emerging trends.

4. Don’t grab shiny new things
While exploring new technologies is encouraged, resist the temptation of adopting every shiny new tool. Avoid "resume-driven development" where you incorporate new libraries or rewrite projects solely for the sake of using the latest technology. Moderation, such as adopting "one new tool per project," is prudent advice.

5. Remember that “skills that got you here won’t get you there”
Recognize that moving to a new career level involves not just deepening technical knowledge but actively applying soft skills. Senior roles require understanding not just how to write code but also knowing what to write and why. Effective communication with people becomes more crucial than interactions with machines at this stage.

What to read for deeper knowledge?

1. Designing Data-Intensive Applications, by Martin Kleppmann
A true classic that delves into the operation of various distributed systems, consensus protocols, SQL and NoSQL databases, and message brokers. While you may not need this depth of knowledge in your daily work, the book provides valuable insights into the stack upon which your systems are built.

2. High Performance Spark / Spark: The Definitive Guide
High Performance Spark — ideal if you're familiar with the basics and seek optimization tips.
Spark: The Definitive Guide — is suitable for those new to the system.

It's advisable to read both books cursorily first, keeping them as references to revisit when needed.

3. Fundamentals of Data Engineering
Geared more towards beginners, this book seems tailored for Data Scientists aiming to grasp the essence of Data Engineering or planning a smooth transition into the role.

4. Software Engineering at Google: Lessons Learned from Programming Over Time
An excellent resource for imbibing best practices in Software Engineering. While not all solutions may directly apply to "Move Fast" startup environments, experienced engineers share valuable insights into designing systems for large-scale and long-term usage.

5. Algorithms and Data Structures for Massive Datasets
Explores probabilistic data structures (e.g., bloom filters, HLL), sampling techniques, and structures optimized for data stored in "external storage." Essential reading for those keen on optimizing the processing of large datasets.

Conclusion

In this article, I have tried to share my expertise in the field of Data Engineering, pointing out the key aspects of career development. I hope that the tips for moving into Data Engineering and using the necessary tools will be useful for those who are looking to reach their potential in this exciting field.

Navigating the Data Engineering Landscape: From Raw Data to Insights

Leo Chashnikov — Mon, 28 Aug 2023 15:34:33 +0000

Probably most readers heard the expression “Data is the new oil”. Crude oil, same as raw data, is much less valuable than its products — petrol in one case, or — insights and understanding that we extract from data.

To be a good source material, our data needs to be accurate and arrive on time, be processed and stored in a way that is easy to discover and query by all the interested parties. Significant part of this process is the task of the Data Engineer.

At the same time, many still confuse Data Engineers with Data Scientists or Software Engineers. So let’s start by defining the differences between them.

Distinctions between Data Engineers, Data Scientists, and Software Engineers

I still consider Data Engineering some sort of Software Engineering “flavor”. It still requires knowledge of one or several programming languages, familiarity with certain (specialized) frameworks, and all the usual computer science algorithms would still be very applicable and beneficial. What differs is much bigger focus on large scale and processing of data using distributed systems (as volumes of data that would require Data Engineers involvement in the process are very unlikely to fit into 1 machine).

And that’s the main difference between a Data Engineer and a Data Scientist. A Data Scientist’s work requires a much deeper understanding of data and business context, math and statistics, but at the same time it’s forgivable for a Data Scientist to be less familiar with good development practices.

Oversimplifying things, one can say that a Data Scientist can produce a Jupyter notebook, that will produce correct results on a data sample, but would be impossible to run in production, and support over a longer period of time. A Data Engineer is expected to take that notebook and turn it into a reliable pipeline, that works with minimal operational demands, and can be easily extended or modified when such need arises.

What tasks usually fall to the Data Engineer’s plate?

First and foremost this list would include ETL (Extract, Transform, Load) Processes and everything around them:

- Data Integration
Collecting data from various sources, such as a diverse spectrum of databases, APIs, and third-party applications, and representing them in a unified and coherent format. They ensure that data is compatible and can be efficiently processed for analysis and reporting. This, of course, implies.

- Data Modelling
Designing data models for storing data within databases or warehouses.

- Data Warehousing
Creating schemas, setting up partitioning strategies, and managing data retention policies to ensure efficient storage and retrieval of data.

- Performance Optimization
Improving data processing speed and keeping its cost under control.

- Monitoring Data Quality
Both pipeline themselves, and data they produce need to be controlled. Pipeline that plainly fails is an obvious problem, but there are many more subtle ones — anomalies in data, corruption, delays, sudden changes in format of upstream data.

- Data Security and Compliance
Implementing security measures to protect data from unauthorized access, ensure compliance with data protection regulations and industry standards.

Lots of those things would require a lot of communication with other stakeholders inside the company — Software Engineers from other teams, who are producers or consumers of data you’re “delivering”, Data Scientists, who are providing algorithms to actually extract insights from data, and all the downstream users, who would be querying results of your work — hence they can tell you about access patterns and use cases.

Necessary skills to get you started

Having a Software Engineer background is a great starting point to becoming a Data Engineer, but it’s not absolutely necessary. Anyway, you wouldn’t go far without at least 1 (the more the merrier) programming language in your toolbelt:

- Python
It would probably be the most obvious choice as it is relatively easy to start with, has very wide-spread utilities and libraries, and as Data Scientists are already using it extensively, it would be easier to talk with them, having common “reference points”.

- Scala
Another good, though more complicated candidate to start with. While pyspark is catching up on most of the features, as Apache Spark is written in Scala, it will still stay a dominant language, allowing one to do more, and with less resources.

- SQL
In addition to programming language, you won’t be able to get far without a good understanding of SQL, as most of warehouses by default support SQL-like data query language (always with some non-obvious confusing differences, that will bite you when you don’t expect), as well as any relational databases, that would often be one of your data sources.

- Apache Spark
To this skillset you’ll need to add some data processing framework. Apache Spark is one of the most popular choices, that provides one with a wide range of capabilities, though it has some competitors depending on specific use case, for instance Apache Flink’s strength is processing of streaming data.

Key knowledge areas for advancement

To navigate further in the realm of enterprise data processing, the foundational toolset covered earlier is just the beginning. As you advance, several critical areas demand your attention:

- Differences between NoSQL and SQL databases
You'll frequently encounter NoSQL databases as sources or destinations in your data pipelines. It's crucial to grasp the distinctions between NoSQL and SQL databases, as their structures, query languages, and use cases differ significantly.

- Lambda architecture and pipeline design
The enduring Lambda architecture, despite its age, remains a key pipeline design pattern. Be prepared to support both batch processing and streaming data pipelines concurrently. Each mode comes with its own intricacies, guarantees, expectations, and challenges.

- Processing frameworks and competitors
While Apache Spark is a robust choice for both batch and streaming processing, alternatives like Apache Flink, Druid, and Kafka Streams also thrive in this landscape. Familiarize yourself with their strengths and applicability.

- Apache Kafka and data streaming
In the realm of streaming, Apache Kafka stands out as the go-to distributed data streaming platform. Its ecosystem of solutions is integral to handling real-time data flows efficiently.

- Workflow management systems
As pipeline dependencies grow complex, employing a workflow management system becomes crucial. Consider a scenario where data from an internal database needs to be integrated with a dataset from an S3 bucket, uploaded by a client once an hour. While the instinct might be to set up a cron schedule, this can lead to problems if the dataset is delayed. Workflow management systems mitigate such issues by allowing data processing only when a clear signal of data completeness is received. Systems like Apache Airflow offer seamless integration of diverse data processing.

- Infrastructure understanding
To ensure the reliable execution of your data pipelines, a deep understanding of the underlying system is necessary. Familiarize yourself with tools like Docker to manage dependencies and create interchangeable and upgradable hosts. Kubernetes is essential for orchestrating hundreds or thousands of hosts where tasks are executed. Additionally, a solid grasp of your chosen cloud provider's offerings is vital.

Growing your career

While this article to that point was mostly dedicated to technical skills, it’s important to remember that purely technical skills can only get you that far. After a certain point soft skills and domain knowledge would be playing a bigger and bigger role.

So here is a list of things, considering which will help to boost your career and grow faster:

- Remember that even as you’re working on internal infrastructure, you still have clients
The fact that they’re employees of the same company doesn’t make them any less valuable. The Data Engineering team should be an enabler for other teams, providing them with data they need to make decisions, and making it easy to access it in the right way.

- Domain knowledge often gives you super power
I cannot count the number of cases where a complex technical solution was replaced with a much simpler, cheaper and more reliable one, as soon as engineers actually understood what users want, not what they seemingly describe.

- Data pipeline connects data from different sources — you connect different teams and align them
Hence the ability to talk to others using their language, and understanding their point of view is crucial.

- Staying up-to-date with the latest trends and technologies
Developing a successful career as a Data Engineer requires a proactive approach that encompasses a range of strategic steps. By staying up-to-date with the latest trends and technologies, you ensure your skills remain relevant in a dynamic industry. Embracing continuous learning keeps you at the forefront of innovation, enabling you to implement cutting-edge solutions and adapt to evolving challenges.

- Participating in data engineering communities and networking events
Engaging with fellow professionals fosters knowledge exchange, idea sharing, and problem-solving collaboration. Through these interactions, you gain insights into best practices, novel techniques, and real-world experiences, enriching your skill set and broadening your perspective.

- Understanding different career paths and roles within data engineering
This knowledge empowers you to make informed decisions about your professional journey. You might choose to specialize in a specific domain, such as machine learning integration, or explore roles like data architect or data scientist. A comprehensive understanding of these options enables you to navigate your career path with clarity and purpose.

- Advancing to leadership positions and broader responsibilities
As you amass experience and expertise, you can transition into leadership roles where you guide teams, make strategic decisions, and shape the direction of data initiatives. This elevation not only showcases your proficiency but also allows you to influence and drive organizational success through data-driven decision-making.

In the realm where data transforms into insights, Data Engineers play a pivotal role by ensuring accurate, timely, and efficient data processing. Armed with technical prowess, effective communication, and an ever-curious mindset, they bridge the gap between waw data and meaningful understanding, driving innovation in the world of data and technologies.