DEV Community: Preeti Hemant

What is Data Profiling?

Preeti Hemant — Sat, 05 Mar 2022 20:42:28 +0000

Imagine this scenario...

You have started a new job as a Data Engineer. It's an exciting time, new data, new tools, new people! While you are familiarizing yourself, your manager who is also very excited to have you look at their backlog, sends a request across.

"One of our stakeholders has been waiting on this for a long time - could you look into this job that's resulting in a very large table in the data warehouse. The stakeholder would love an aggregated table with the data rolled up to a week."

What is one of the first things you do?

Data Profiling!

It's a systematic analysis of the data in a table to understand the structure and relationships - leading to conclusions on its usability.

The tools to extract this information are measures in descriptive statistics - min, max, mode, frequency, sum, count (You get the picture).
The next set of parameters to understand are data types, nulls, uniqueness which answer questions on missing data, duplicates and so on.

You could run this analysis on individual columns or do a cross-columns analysis.

At the end of this exercise, you should be able to make a few decisions
1) Is this data able to support the analysis/reporting use case?
2) Do the data quality issues originate from the source or from the ETL jobs?
3) Are there inconsistencies in the structure of the table?

So, this was a short introduction to the Data Profiling technique. For further reading, here are some links.

Data architecture models

Preeti Hemant — Wed, 23 Feb 2022 03:11:53 +0000

Data has come a long way, starting from the 1640s when the term “data” had its first use, to the 21st century, where AI has become integral to everyday life.

As you can imagine, several software and hardware developments have co-evolved with data, bringing us to the here and now. One of the early challenges in data was ingesting it — how the data was to be used and the needs it served, weren't nearly as interesting. The use cases were extremely narrow, mostly defaulting to basic business reporting. Today however, the focus has shifted from ingesting data to making it accessible in a way that would support a plethora of applications — parameters of accuracy, timeliness, reliability and trust at a massive scale, are paramount.

The challenges today in data - a consequence of its scale and speed - are in the areas of data discoverability, governance and reliability. The market is flooded with tools for every data problem conceivable. But, is there a guiding philosophy on how to bring these multitude of tools together, or how to stitch the different roles in an org with these tools?

What we need are data architectures that can provide directional guidance, allow for weighing trade-offs, are domain-agnostic and at the same time don’t put us at the risk of building something that quickly becomes obsolete.

Data architecture is a relatively new term. In fact, one of the first references to data architecture is the mention of Data mesh as a model in this article in April, 2020.

So, is data mesh the only model or one of many? A search will show "Data Fabric" and "Data Mesh" as two popular candidates for data architecture models.

If you are looking for a short form introduction to the two models; Think of data fabric as a convergence of the modern data tools, stitched together to collect disparate data and move it within a system in a multi-hop manner. The objectives being, data discoverability, accessibility and management — for varied consumers and use cases. Data Mesh then, is the next step in the evolution of data architectures; brining in aspects of product management and decentralization to data.

Contrasting one with the other, Data fabric allows for ingestion of data from any source, for any use case — without gating it for quality during ingestion — trust in data, data integrity are addressed through layers that logically come after ingestion. Whereas, Data mesh places strong emphasis on data quality and data being treated as a product, even before it can become part of the data ecosystem.

Is one better than the other? Which one renders itself better to implementation? Here’s the long form of the two models.

Data Fabric

How should data move in a system, What characteristics should data retain and shed as it moves? In the Data fabric architecture, data follows a set of steps that determine its flow. The first step takes data through an integration phase. In the integration phase, data is ingested and then cleaned, transformed and loaded into storage. Then, there is the data quality phase where quality assessment is performed on the stored data. This data is then made available for different use cases through a combination of a data lake and a data warehouse, Typical use cases are BI, analytics and machine learning. Data governance policies are defined for the ingested data and a data catalog is used for discoverability.

The above functions are mostly centralized — a team of data specialists are designing and implementing the different stages in the fabric and also setting up policies and access controls.

Simply put, Data Fabric is how most data ecosystems move, store and access data today.

The beauty of data fabric as an architecture model is the flexibility it offers — not all components are a must, there are multiple vendors with off-the-shelf solutions that can collect and process data from any source and any use case.

Data Mesh

Riding on the shift of software systems towards distributed domain design, data mesh is built on the principles of distributed architecture. There are three major components in a data mesh - Decentralized Domain ownership of data and the resulting Data products, Self-serve data infrastructure and Federated Governance.

Data Mesh has been designed to derive value from data at scale, in complex environments — complex not in data volume or velocity but in the number of use cases and the diversity of data sources. Since the complexity in not only technical, this architecture is modelled as a socio-technical construct.

Domain owned data is probably the most critical shift in going from Data Fabric to a Mesh. The idea is quite simple — Who better to own and provide data for use, than the teams generating the very data? In this paradigm, business domains decide what data is useful and should be exposed for different use cases within the org. If that is true, are these the teams also building methods and tools to serve this data? No — This requires skills that the domains are not expected to have and is instead delegated to the data infrastructure that builds a self-serve data platform.

Domains serve their data as a product — a product that meets well-defined standards that ensure interoperability with data from other domains. This data product lives as a node on the mesh. This is how the concept of ETLs is done with in the Data Mesh paradigm.

Decentralized domain data ownership is the highlight in this architecture. Ownership of design and deployment of the infrastructure that serves data, however is centralized — with the data platform team. Naturally, there arises a need for a body that balances these aspects, delineates decisions that lie localized with each domain from the decisions that are considered global. This group is the federated governance group that is carved out of both the data platform team and individual domains.

Similarities between Data Fabric and Data Mesh
Both the architecture models attempt to solve the problem of getting value from data at scale - while making data secure, accessible and easy to use and interpret.

How do they differ?
In a Data Fabric, a dataset gains value by being onboarded, catalogued and made available through a standardized set of governance rules.
In a Data Mesh, a dataset gains value because of its usability as determined by its consumers (data scientists/data analysts)

In a Data fabric, there is standardization in how data is cleaned, labeled and checked for quality.
In a Data Mesh, the decision on how data is to be made consumption ready i.e the pre-processing steps lies with the domains that own the data.

In a Data Fabric, the onus of understanding the data, interoperability of data sets generated by different services becomes a joint responsibility of the data engineering team and consumers of data - the analysts and the scientists.
In a Data Mesh, it is the responsibility of the teams serving their data, to understand how the data could be used to generate value and design it in a way that meets the needs of the consumers.

Finally, which one should you pick?
Data Fabric addresses and recommends solutions to the fundamental questions on ingestion and use of data. Data Mesh as a model, can become a solution when the fabric hits a wall on issues around data ownership and data quality. Also, an important pre-requisite for Data Mesh architecture to be successful is domain oriented software architecture and teams in an organization.

All things considered, it is a good idea for a data org to get started with the data fabric paradigm and adopt principles from data mesh as their data, their needs and complexity of the data systems evolve!

References:
Data Mesh Principles and Logical Architecture
HelloFresh Journey to the Data Mesh
Data Fabric as Modern Data Architecture

12 Rules for Recruitment

Preeti Hemant — Sat, 08 Jan 2022 19:33:15 +0000

Hiring can be a hit or a miss. While there is no set formula to hiring good candidates, there are some practices that can be used set up a good hiring process!

Here are 12 "rules" that have helped me grow my teams.

Attract the right candidate
One of the first steps to hiring is creating a pool of good candidates by gaining attention of suitable candidates. An accurate job description, an outline of the day-to-day work and expected outcomes at the three, six and twelve month mark, attract candidates with relevant experience and interests. During the recruiter screening phase, sharing details on the interview steps and the onboarding plan (created for new hires) further provide clarity and help keep candidates in the interview loop.
Develop a bias towards a short interview loop
A quick and efficient interview process is highly desirable, both by candidates and hiring teams. Lengthy and involved interviews tend to push candidates away. It also means a continued engagement from your team. While it is not easy to contain all aspects of evaluation in just a couple of rounds, it is possible to trim redundancy and tighten the process.
Design the early rounds of interviews to test for well-roundedness
With a bias to shorter interviews, it is critical that early rounds test for a variety of traits and skills. An interview that evaluates cultural fit through value-based questions gives a peek into a candidates' thought processes and mindset.
Screening for the desired attitude and pre-dispositions early on, can be valuable.
Define an ideal candidate profile
Each one of the interview rounds help build a perceived profile of a candidate. Having a reference that defines what an ideal candidate looks like, helps keep the evaluation accurate, fair and unbiased.
Recruit a diverse interview panel
A hiring manager and the team may best understand the requirements of the role, but only a diverse panel can judge the many aspects that make someone successful at work.
Include technical and non-technical folks in the interviews.
Provide a sneak peek into the team, org culture and upcoming projects
Recruitment is always a two-way street - It is important that both the team and candidates see the role as a good fit. Candidates understand what it takes to flourish in your team, if the culture is a match and they see themselves succeeding in the team. I have found that sooner an applicant meets the hiring manager, the better it is to address and discuss team culture and goals.
Involve the team and set up candidate-team interactions
Getting the team to meet the candidate and vice versa ensures the candidate can validate the culture as described by the hiring manager, it is also an opportunity to build consensus on the candidate within the team. If your team is large with many members, then it is important that at least the people expected to interact the most with the potential hire meet before hand.
Explain the role of the team and its importance in the org
Everybody is looking to make an impact through their work. Let candidates have clarity on how their work can create an impact from within the team. For this, the role of your team in the org and how it contributes to business goals is an important detail to share and discuss.
Explain Org Goals and Strategy
Organizational strategy and business goals create excitement and help candidates picture their growth. It gives them a chance to evaluate if the org strategy aligns with their long term interests and to see a future in the organization.
Provide details on tech stack, tech maturity and complexity of the team's tasks
Candidates rarely are an ideal fit. Most applicants have experience in a different set of processes and practices. Discussing these details allow both the hiring manager/team and the candidate to compare and contrast methods and approaches. It helps with identifying gaps. It also helps with assessing parts of their experience that can be leveraged and skills that are transferrable.
Don't rely solely on ATS
There are too many examples of recruitment gone wrong due to the errors and biases introduced by automation. My favourite is the fiasco AI assisted ATS created for Amazon.
Human-in-the-loop screening is time consuming, but in many cases worth the effort. In my experience, there are many variations in documents that candidates submit and screening is best not left to the ATS alone.
Iterate on the 11 steps mentioned above
The most important of all rules, perhaps, is the last one - Iterate on your hiring process and strategy. No two roles are the same and no two times are the same.
Adapt and adopt!

Hiring - Candidate Personas

Preeti Hemant — Thu, 02 Dec 2021 08:31:17 +0000

What candidate persona would add the most value to a team?

Borrowing from the game of football, a strategy of hiring squad members and a playmaker is ideal for setting up and growing a team.
The two roles become critical in a team that is looking to balance sustenance activities along with innovation.

So, how does one identify these qualities in candidates and build them in team members?
First, let's look at these two personas individually.

Squad member
Squad members perform in an individual capacity to deliver outputs on clearly defined requirements.

Their responsibilities mostly include

Effectively delivering on individual tasks
Implementation of small projects

Look for hiring indicators and values like Efficient delivery, Well rounded execution of tasks, Communication, Collaboration, Code fluency and Empathy

A squad member has

The abilities to perform day to day individual tasks well (effectiveness is experience dependent)
Clear and effective communication
Dedication to continuously learn and improve
Problem solving abilities (Nice to have)
Good time management (Nice to have)
Self Management (comes with experience)

Playmaker (Expert member)
These members are the creative force behind a team, they architect solutions and build them by engaging other squad members.

They can deliver on a diverse set of responsibilities like

Building solutions that are modular, scalable and forward looking
Effectively delivering projects through a large team with significant complexity
Training/coaching/mentoring team members to deliver on smaller tasks that come together as a larger solution

Look for hiring indicators and values like Forward looking, Strategic thinking, Critical thinking, Strong sense of ownership, Decision making, Passion for knowledge sharing, Creative problem solving, Team health, System Design, Empathy(very important)

A playmaker has

The ability to work with nebulous asks and translate them into well defined smaller tasks
Proven track record in architecting solutions that solve problems and address shortcomings in the near, mid and long term
Understands the importance of coaching/mentoring squad members and facilitates knowledge distribution

In the long term, they raise the bar for the entire team.

Finally, the distribution of squad and expert members is dictated by the nature and scope of the work in a team, more specifically the ratio of sustenance to new and innovative work.

In most situations, I have found this is to be an effective hiring strategy, with case-specific tweaks.

What is yours? Please comment and share!