DEV Community: Bala Priya C

A Step-by-Step Starter Kit for Building a Data Quality Framework

Bala Priya C — Thu, 07 May 2026 04:39:06 +0000

There is no shortage of frameworks for thinking about data quality. There is, however, a significant shortage of practical guidance for actually building one from a standing start. This is especially true for teams that don't have years to spend on the project and need to show value quickly while building toward something sustainable.

This guide is written for that. It assumes you have organizational support for a data quality initiative, some existing data infrastructure, and people who care about getting this right, but not necessarily a large dedicated team or a clear playbook. It is sequenced so that each step produces something useful before the next begins.

Step 1: Define What "Quality" Means for Your Organization

Before you measure anything, answer this question: quality for what purpose? Data quality is not an abstract property of data; it is always relative to a use case. Data that is perfectly adequate for internal trend analysis may be inadequate for regulatory reporting. Data that meets the requirements of a batch analytics process may be too stale for a real-time operational system.

Spend time in this step interviewing the primary consumers of your most important data assets. Ask them:

What data do you depend on most heavily?
When has data quality caused you a problem in the past year?
What would you need to know about a dataset to trust it?
What are the consequences when data quality fails?

The output of this step is a written statement of quality requirements for your most critical use cases, not a generic checklist, but specific, use-case-tied requirements. This document will drive everything that follows.

Step 2: Identify and Prioritize Your Critical Data Elements

You cannot govern everything equally. Trying to do so is one of the most common reasons data quality programs stall. The scope becomes so large that progress on any individual area is imperceptible, and the program loses momentum before it achieves anything demonstrable.

Critical data elements or CDEs are the fields and datasets that matter most to the business: those that feed key reports, influence material decisions, or carry regulatory risk. Identifying them requires input from both business stakeholders and data engineers who understand downstream dependencies.

A practical approach: ask each business domain to nominate the five to ten fields they most depend on, then cross-reference with the data lineage of your highest-priority reports and dashboards. The intersection of "business-critical" and "frequently used" is a reasonable starting point.

Prioritize well. A working quality framework for twenty CDEs is more valuable than a nominal framework for two hundred.

Step 3: Baseline Current Quality

Before you can improve quality, you need to know where you are. This step involves profiling your CDEs against the quality requirements you defined in Step 1 — not just running generic completeness and uniqueness checks, but assessing against the specific standards that matter for each use case.

Document what you find honestly. It is common to discover that quality for important fields is significantly worse than expected. That is useful information, not a failure. It demonstrates the value of the initiative and identifies where to focus remediation effort.

The baseline serves two purposes. It gives you a starting point against which future improvement can be measured. It also gives you the business case for the governance investments that follow.

Step 4: Establish Ownership

Data quality without clear ownership is an aspiration, not a program. For each CDE, there should be a named owner: a person who is responsible for the quality of that data, who understands what the field represents and how it is produced, and who has the organizational standing to drive corrective action when quality fails.

Ownership is often the most (politically) difficult step. In many organizations, no one wants to be named as accountable for something that is frequently broken. This friction is informative because it tells you where governance gaps exist.

Work through it explicitly. Engage business unit leaders in the ownership assignment process. Make clear that the owner's responsibility is to participate in quality improvement, not to be personally blamed for historical failures. Create a lightweight accountability structure, like a regular review, a clear escalation path, that makes ownership manageable rather than burdensome.

Step 5: Define and Implement Quality Rules

For each CDE, translate the quality requirements from Step 1 into specific, testable rules.

Completeness: what percentage of records must have a value in this field?
Validity: what values are permissible?
Consistency: where the same data appears in multiple systems, how closely must the values match?
Timeliness: how current must the data be for its use case?

Implement these rules in your existing tooling, whether that is a dedicated data quality platform, dbt tests, Great Expectations, or custom SQL checks. The goal at this stage is not perfection but coverage: every CDE should have at least the most critical rules implemented and running on a defined schedule.

Document each rule, its business rationale, and its owner. Rules without documented rationale get disabled during pipeline refactoring because no one knows why they exist.

Step 6: Build the Response Process

Automated quality checks are inputs, not solutions. When a rule fails, something has to happen: the failure is investigated, the cause is identified, a fix is implemented, and the stakeholders affected are informed.

This requires defining the response process in advance, including how failures are handled, who gets notified, expected response times by severity, who can decide to hold back a data product, and who tracks resolution.

This process is the difference between a quality monitoring system and a quality improvement system. Organizations with the former know about their quality problems. Organizations with the latter fix them.

Step 7: Communicate, Report, and Iterate

Data quality cannot improve in isolation from the people who depend on it. Build a communication cadence into the framework: regular reports to business stakeholders on quality status for their CDEs, clear channels for reporting issues that aren't caught by automated checks, and transparency about known limitations.

Establish a review cycle — quarterly works well for most programs — where the framework itself is assessed. Try to answer the following questions systematically:

Which rules are catching real problems?
Which ones are generating noise?
Which CDEs need coverage expansion?
Which quality failures repeated in the last quarter, and what does that tell you about the root cause?

Iteration is not a failure of the framework. It is how a framework matures. The version you deploy in month one should look different from the version running in year two because you learned things from using it.

Conclusion

Here's a summary of the steps outlined:

Step	Focus Area	Key Actions	Output	Why It Matters
1	Define Quality	Interview stakeholders, identify use cases, clarify expectations	Use-case-specific quality requirements	Ensures quality is tied to real business needs
2	Prioritize CDEs	Identify and shortlist critical data elements (CDEs)	Focused list of high-impact data fields	Prevents scope overload and enables quick wins
3	Baseline Quality	Profile data against defined requirements	Current quality assessment	Establishes starting point and highlights gaps
4	Establish Ownership	Assign accountable owners for each CDE	Named data owners + accountability model	Enables responsibility and action on issues
5	Implement Rules	Define and deploy validation rules (completeness, validity, etc.)	Automated quality checks with documentation	Turns requirements into enforceable controls
6	Response Process	Define workflows for handling rule failures	Incident response process + SLAs	Ensures issues are resolved, not just detected
7	Communicate & Iterate	Report status, gather feedback, review regularly	Continuous improvement cycle	Keeps framework relevant and effective over time

A realistic timeline for a starter framework covering twenty to thirty CDEs is about four to six weeks for scoping, prioritization, and baselining, followed by another four to six weeks for ownership and rule implementation, and two to four weeks to establish the response process. By week sixteen at the latest, you should have a working setup that delivers real value.

That is fast enough to maintain organizational momentum and demonstrate that the investment is worthwhile. It is also ambitious enough to require discipline — which is why the prioritization in Step 2 is not optional.

Start narrow, deliver demonstrable value, build organizational trust in the program, then expand scope. That sequence succeeds far more often than trying to build the complete framework before showing results.

7 Signs Your Data Quality Framework Is Broken

Bala Priya C — Wed, 06 May 2026 01:54:45 +0000

Most organizations have some version of a data quality framework. Fewer have one that works. The gap between having a framework and having a functioning one is wide, and it tends to widen quietly — through gradual neglect, accumulated technical debt, and the slow erosion of accountability that happens when no one is specifically paid to care.

Here are seven signs that your data quality framework has stopped doing its job and what each one tells you about the underlying problem.

1. Your Quality Metrics Are Green but Business Complaints Are Constant

This is the most common pattern and the most diagnostic. The data quality dashboard shows acceptable or good scores across completeness, accuracy, and timeliness dimensions. Meanwhile, the business analytics team regularly flags data problems, analysts spend significant time cleaning data before they can use it, and reports occasionally publish numbers that don't match reality.

When this gap exists, the framework is measuring the wrong things or measuring them at the wrong level of granularity. Technical completeness metrics, for instance, confirm that a field is populated. They do not confirm that the value is correct, that it was populated using consistent logic, or that it means the same thing across systems.

The fix is to work backward from actual business pain.

Document recent data quality failures that caused business impact.
Ask what metric, if it had been in place, would have caught each failure.

Build your measurement framework from that list, not from a generic data quality model.

2. No One Can Name the Owner of a Quality Problem

When a data quality issue surfaces, what happens? If the answer involves a period of investigation to determine whose domain the data belongs to, followed by a negotiation about whether the issue is a source system problem or a transformation problem, followed by someone filing a ticket that ages in a queue, the framework has an accountability gap.

Quality frameworks require explicit ownership. Not nominal ownership listed in an RACI document, but operational ownership: a named person with clear responsibility for defined datasets, the authority to take corrective action, and a defined process for escalation when the issue exceeds their scope.

If your framework does not have this, quality issues will be discovered, discussed, and inadequately resolved. The same categories of problems will recur because there is no one whose job it is to prevent recurrence.

3. Rules Are Defined But No One Reviews Exceptions

Many organizations have invested in data quality tooling that runs automated checks and generates exception reports. The checks run. The exceptions accumulate. The reports sit unread, or are reviewed by someone who acknowledges them and moves on.

Automated quality checks are inputs to a process, not the process itself. They require a human review loop with defined response thresholds:

Exception rates above X trigger investigation
Recurring exceptions trigger root cause analysis
Systemic failures trigger escalation to data governance.

Without that loop, you are generating increasingly accurate data about your quality problems and doing nothing with it.

Review the volume of unaddressed exceptions in your current quality tooling. If it is large, the framework has a process gap between detection and resolution.

4. Quality Standards Were Set Once and Never Revisited

Data quality standards should reflect use case requirements. A field that feeds a regulatory report has different accuracy requirements than one that feeds an internal exploratory dashboard. The appropriate timeliness standard for a real-time pricing engine is different from that for a monthly financial summary.

Frameworks that define quality standards once, at implementation, tend to drift out of alignment with actual use cases as the business evolves. New data products get launched that inherit standards that weren't designed for them. Thresholds that were appropriate for historical query volumes become inadequate when data starts feeding machine learning models.

Quality standards should be reviewed whenever a dataset acquires a significant new use case. They should also be reviewed periodically — at minimum annually — to confirm that they still reflect how the data is being used and what failures would actually cost.

5. Downstream Systems Have Their Own Duplicate Quality Checks

When the analytics team is running their own validation scripts before using data, when the reporting team has a set of "sanity checks" they run on every extract, when multiple teams have independently built processes to detect the same categories of problems — that is a signal that the central quality framework is not trusted.

Each of those independent checks represents rework. It also represents risk, because teams are applying different standards and definitions, which means the same underlying data can produce different results depending on who processed it. When those results eventually collide — in a presentation, a regulatory filing, a cross-functional analysis — it creates more damage than the original quality problem would have.

The duplication problem is usually a trust problem. Rebuilding trust requires transparency about quality status, consistent resolution of reported issues, and reliable communication when something has changed.

6. Data Quality Is Treated as a Data Team Problem

In healthy data quality programs, business stakeholders are active participants. They define what good looks like for their use cases. They report issues through a clear channel and receive timely responses. They participate in root cause analysis for significant failures. They understand that some quality problems originate in the business processes that generate the data, not in the data systems that store it.

When business stakeholders have opted out — when they view data quality as a technical problem they don't need to engage with — the framework is missing a critical input. The data team cannot fully understand the business impact of quality failures without that input. And quality problems that originate in business processes, like inconsistent data entry practices, cannot be fixed from the data layer alone.

Reconnecting business stakeholders requires demonstrating that their participation produces better outcomes for them — faster resolution of the issues that affect their work, data they can trust without running their own checks.

7. There Is No Feedback Loop From Production Issues to Prevention

The most mature quality frameworks treat production issues as learning opportunities. When a quality failure causes a business problem, the response includes not just remediation but root cause analysis and process improvement. The knowledge gained from the failure is used to build better checks, close governance gaps, and prevent the category of problem from recurring.

Organizations without this feedback loop fix the same types of problems repeatedly. The fixes are reactive, the root causes persist, and the true cost of poor data quality — in engineering time, business disruption, and eroded trust — compounds over time.

Building the feedback loop requires designating someone responsible for it — typically within the data governance function — and creating a regular cadence for reviewing quality incidents, documenting patterns, and translating those patterns into framework improvements.

Final Thoughts

If several of these signs are present, the problem is not technical. It is structural. No amount of tooling investment will fix a framework that lacks clear ownership, genuine accountability, and regular review processes.

The technical tools serve the framework. The framework requires human commitment to function. Even one small improvement that is actually implemented is worth more than a comprehensive framework that exists on paper.

Descriptive Statistics with Python for Beginner Data Scientists

Bala Priya C — Wed, 11 Mar 2026 07:37:03 +0000

You've probably heard that data scientists spend most of their time cleaning and exploring data... not building fancy models. That's true, and descriptive statistics is the foundation of that exploration.

Before you can ask "what does this data predict?", you need to ask "what does this data look like?" Descriptive statistics gives you the tools to answer that. It tells you where your data is centered, how spread out it is, and whether it's shaped in a way that might cause problems for the models you'll build later.

In this article, you'll work through the core concepts of descriptive statistics using Python, pandas, and matplotlib. Along the way you'll build intuition — not just know which function to call, but understand what the numbers are actually telling you.

If you'd like, you can follow along with the Google Colab notebook.

Prerequisites

Before reading this article, you should be comfortable with:

Basic Python (variables, lists, loops, functions)
What a DataFrame is in pandas (you don't need to be an expert; knowing pd.DataFrame() and df.head() is enough)
Installing packages with pip

You do not need any prior statistics knowledge. We'll build everything from scratch.

What Is Descriptive Statistics?
The Dataset
Measures of Central Tendency — Mean and Median
Measures of Spread — Std Dev, Variance, IQR
The Five-Number Summary
Skewness — Is Your Distribution Symmetric?
Putting It All Together
Which Chart for Which Job?
Summary

What Is Descriptive Statistics?

Descriptive statistics is the practice of summarizing and describing the features of a dataset. Unlike inferential statistics — which draws conclusions about a larger population from a sample — descriptive statistics just tells you about the data you actually have in front of you.

Think of it this way: if a colleague hands you a spreadsheet of 10,000 sales transactions and asks "what do you see?", you can't read every row. Descriptive statistics is how you compress that spreadsheet into a handful of numbers that tell the real story.

There are three categories of descriptive statistics you'll use often:

Category	What it answers	Examples
Central tendency	Where is the "middle" of my data?	mean, median, mode
Spread / variability	How spread out is the data?	variance, std dev, range, IQR
Shape	What does the distribution look like?	skewness, kurtosis

Let's work through each one.

The Dataset

We'll use sales records from 12 retail store branches across a single quarter. Each row represents one branch's performance: total revenue, units sold, and product return rate.

This is small enough to reason about by hand but realistic enough to make the statistics meaningful.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

data = {
    "branch": ["Atlanta", "Boston", "Chicago", "Dallas",
               "Denver", "Houston", "Miami", "Nashville",
               "Phoenix", "Portland", "Seattle", "Tampa"],
    "revenue_usd": [142000, 198000, 175000, 161000,
                    134000, 189000, 210000, 123000,
                    155000, 168000, 202000, 139000],
    "units_sold": [940, 1280, 1105, 1020,
                   870, 1190, 1340, 790,
                   985, 1060, 1295, 855],
    "return_rate_pct": [3.1, 2.4, 2.9, 4.2,
                        3.8, 2.1, 1.9, 5.1,
                        3.4, 2.7, 2.2, 4.6]
}

df = pd.DataFrame(data)
print(df.head())

Output:

    branch  revenue_usd  units_sold  return_rate_pct
0  Atlanta       142000         940              3.1
1   Boston       198000        1280              2.4
2  Chicago       175000        1105              2.9
3   Dallas       161000        1020              4.2
4   Denver       134000         870              3.8

Measures of Central Tendency

Central tendency tells you the "typical" value in your data — the value around which everything else clusters.

Mean

The mean is the sum of all values divided by the count. It's the most commonly used measure of center, but it has a weakness: it is sensitive to outliers.

If one branch had a super good quarter at $900,000, the mean would shoot up and no longer represent the typical branch.

Median

The median is the middle value when your data is sorted. Half the values fall below it, half above.

Because it only cares about position — and not magnitude — extreme values don't move it. This makes it more robust than the mean.

A useful rule of thumb: if mean ≠ median, something interesting is going on. It could be outliers, skewness, or both.

Let's compute both:

revenue = df["revenue_usd"]

mean_rev   = revenue.mean()
median_rev = revenue.median()

print(f"Mean revenue:   ${mean_rev:,.0f}")
print(f"Median revenue: ${median_rev:,.0f}")

Output:

Mean revenue:   $166,333
Median revenue: $164,500

The two values are close — only $1,833 apart. That tells us no single branch is substantially distorting the average. The data is reasonably balanced.

Visualizing Mean vs Median

A bar chart with reference lines is the clearest way to see this relationship in action. Bars below the mean are colored blue, above it green, so you can immediately see which branches are performing above and below average.

fig, ax = plt.subplots(figsize=(10, 5))

colors = ["#4a90d9" if v < mean_rev else "#2ecc71"
          for v in df["revenue_usd"]]

ax.bar(df["branch"], df["revenue_usd"] / 1000,
       color=colors, edgecolor="white", linewidth=0.8)

ax.axhline(mean_rev / 1000, color="#e74c3c", linewidth=2,
           linestyle="--", label=f"Mean: ${mean_rev/1000:.0f}k")
ax.axhline(median_rev / 1000, color="#f39c12", linewidth=2,
           linestyle="-.", label=f"Median: ${median_rev/1000:.0f}k")

ax.set_title("Q3 Revenue by Branch with Mean and Median",
             fontsize=14, fontweight="bold")
ax.set_xlabel("Branch")
ax.set_ylabel("Revenue (USD thousands)")
ax.legend()
ax.tick_params(axis="x", rotation=45)
plt.tight_layout()
plt.savefig("revenue_central_tendency.png", dpi=150)
plt.show()

Output:

In the above dataset, Nashville is the branch with the lowest sales. Miami, Boston, and Seattle lead. The two reference lines sitting almost on top of each other confirms: no outlier is pulling the average off-center.

Measures of Spread

Two datasets can have identical means and still look completely different. One might have all values clustered tightly around the center; another might be all over the place. Measures of spread capture that difference.

Standard Deviation

Standard deviation tells you the typical distance of a data point from the mean. A small std dev means values are packed tightly together. A large one means they're spread out widely.

It's expressed in the same units as your data — dollars here — which makes it easy to interpret directly.

Variance

Variance is the square of the standard deviation. It's used heavily in statistical formulas behind the scenes, but its units are squared (dollars²), which makes it awkward to communicate. Day-to-day, stick with std dev.

Range and IQR

The range (max − min) is simple but not quite robust because one extreme value changes it completely.

The IQR (interquartile range) is the distance between the 25th and 75th percentiles. It describes the spread of the middle 50% of your data. Because it ignores the top and bottom 25%, it's resistant to outliers — just like the median.

std_rev   = revenue.std()
var_rev   = revenue.var()
range_rev = revenue.max() - revenue.min()
iqr_rev   = revenue.quantile(0.75) - revenue.quantile(0.25)

print(f"Std deviation: ${std_rev:,.0f}")
print(f"Variance:      ${var_rev:,.0f}")
print(f"Range:         ${range_rev:,.0f}")
print(f"IQR:           ${iqr_rev:,.0f}")

Output:

Std deviation: $28,908
Variance:      $835,696,970
Range:         $87,000
IQR:           $50,000

The typical branch deviates from the mean by about $29k. The middle 50% of branches span a $50k revenue range. These are the numbers you'd quote in a report. The variance of $831 million — while mathematically correct — would confuse anyone you hand it to.

Visualizing Spread: The Box Plot

The box plot helps visualize the spread of the data. It encodes five numbers in one chart: the minimum, Q1, median, Q3, and maximum.

The box represents the IQR.
The line inside the box is the median. Points beyond the whiskers are potential outliers.

fig, axes = plt.subplots(1, 3, figsize=(12, 5))

cols   = ["revenue_usd", "units_sold", "return_rate_pct"]
labels = ["Revenue (USD)", "Units Sold", "Return Rate (%)"]
colors = ["#4a90d9", "#2ecc71", "#e67e22"]

for ax, col, label, color in zip(axes, cols, labels, colors):
    bp = ax.boxplot(df[col], patch_artist=True, widths=0.5,
                    medianprops=dict(color="black", linewidth=2))
    bp["boxes"][0].set_facecolor(color)
    bp["boxes"][0].set_alpha(0.7)
    ax.set_title(label, fontsize=12, fontweight="bold")
    ax.set_xticks([])

plt.suptitle("Spread of Key Metrics Across Branches",
             fontsize=14, fontweight="bold", y=1.02)
plt.tight_layout()
plt.savefig("boxplots_spread.png", dpi=150)
plt.show()

Output:

The Five-Number Summary

The five-number summary — min, Q1, median, Q3, max — gives you a compact, complete picture of a column's spread. In pandas, describe() gives you this plus the mean and standard deviation for every numeric column at once.

Make this the first thing you run on any new dataset:

print(df[["revenue_usd", "units_sold", "return_rate_pct"]].describe().round(2))

Output:

       revenue_usd  units_sold  return_rate_pct
count        12.00       12.00            12.00
mean     166333.33     1060.83             3.20
std       28908.42      184.29             1.04
min      123000.00      790.00             1.90
25%      141250.00      922.50             2.35
50%      164500.00     1040.00             3.00
75%      191250.00     1212.50             3.90
max      210000.00     1340.00             5.10

A few things stand out:

The return rate range (1.9% to 5.1%) is wide relative to its mean of 3.2%
Revenue mean ($166k) and median ($164.5k) are close — no major outlier distortion
The count row is your quick data quality check: if any column shows fewer rows than expected, you have missing data

Skewness

Skewness measures whether your distribution is symmetric around the mean or has a longer "tail" pulling to one side.

Skewness ≈ 0 — roughly symmetric; the normal distribution is the classic example
Skewness > 0 — right-skewed; a few very high values stretch the tail to the right
Skewness < 0 — left-skewed; a few very low values stretch the tail to the left

Why does this matter? Many statistical tests — and some machine learning algorithms — assume your data is roughly symmetric. If it isn't, results can be unreliable. Knowing skewness early lets you decide whether to transform the data before going further.

for col in ["revenue_usd", "units_sold", "return_rate_pct"]:
    skew = df[col].skew()
    print(f"{col:<25} skewness = {skew:.3f}")

Output:

revenue_usd               skewness = 0.095
units_sold                skewness = 0.167
return_rate_pct           skewness = 0.549

Visualizing Skewness: The Histogram

Histograms are the best way to see skewness. The shape of the bars tells you where data clusters and where it trails off. Overlaying the mean and median makes the direction of any skew immediately visible — when they're apart, the tail is on the side of the mean.

fig, axes = plt.subplots(1, 3, figsize=(13, 4))

cols   = ["revenue_usd", "units_sold", "return_rate_pct"]
labels = ["Revenue (USD)", "Units Sold", "Return Rate (%)"]
colors = ["#4a90d9", "#2ecc71", "#e67e22"]

for ax, col, label, color in zip(axes, cols, labels, colors):
    ax.hist(df[col], bins=6, color=color, edgecolor="white",
            linewidth=0.8, alpha=0.85)

    mean_val   = df[col].mean()
    median_val = df[col].median()
    ax.axvline(mean_val,   color="#e74c3c", linestyle="--",
               linewidth=1.8, label="Mean")
    ax.axvline(median_val, color="black",   linestyle="-.",
               linewidth=1.8, label="Median")

    skew_val = df[col].skew()
    ax.set_title(f"{label}\nskew = {skew_val:.3f}",
                 fontsize=11, fontweight="bold")
    ax.legend(fontsize=9)

plt.suptitle("Histograms with Mean vs Median — Detecting Skewness",
             fontsize=13, fontweight="bold", y=1.03)
plt.tight_layout()
plt.savefig("histograms_skewness.png", dpi=150)
plt.show()

Output:

Look at the return rate panel on the right. The mean (red dashed line) sits to the right of the median (black dash-dot). That gap is the visual fingerprint of positive skew. Revenue and units sold, by contrast, are not as skewed.

Putting It All Together

Here's a reusable function that prints the key descriptive stats for any numeric column:

def summarize(series, label):
    print(f"\n── {label} ──")
    print(f"  Mean:    {series.mean():>10.2f}")
    print(f"  Median:  {series.median():>10.2f}")
    print(f"  Std Dev: {series.std():>10.2f}")
    print(f"  IQR:     {series.quantile(0.75) - series.quantile(0.25):>10.2f}")
    print(f"  Skew:    {series.skew():>10.3f}")

summarize(df["revenue_usd"],     "Revenue (USD)")
summarize(df["units_sold"],      "Units Sold")
summarize(df["return_rate_pct"], "Return Rate (%)")

Output:

── Revenue (USD) ──
  Mean:     166333.33
  Median:   164500.00
  Std Dev:   28908.42
  IQR:       50000.00
  Skew:         0.095

── Units Sold ──
  Mean:       1060.83
  Median:     1040.00
  Std Dev:     184.29
  IQR:         290.00
  Skew:         0.167

── Return Rate (%) ──
  Mean:          3.20
  Median:        3.00
  Std Dev:       1.04
  IQR:           1.55
  Skew:         0.549

Drop this function into any exploratory notebook and run it on every column before you analyze the data further.

Which Chart for Which Job?

Here's an overview of which chart you should use:

What you want to show	Best chart	Key method
Distribution shape and skewness	Histogram	`ax.hist()`
Spread, median, and outliers	Box plot	`ax.boxplot()`
Comparing values across categories	Bar chart	`ax.bar()`
Mean vs median on a bar chart	Bar + reference lines	`ax.axhline()`

Summary

Concept	What it tells you	pandas method
Mean	Average value	`.mean()`
Median	Middle value — outlier-robust	`.median()`
Std deviation	Typical distance from the mean	`.std()`
IQR	Spread of the middle 50%	`.quantile()`
Skewness	Symmetry of the distribution	`.skew()`
Five-number summary	Full spread at a glance	`.describe()`

What's Next?

In this article you learned to describe your data with numbers and charts. But there's a deeper question behind all of this: why do values cluster and spread the way they do? The answer lies in probability distributions, and that's exactly what we'll cover in the next article in this series.

How to Work with Parquet Files in Python – A Guide with Examples

Bala Priya C — Thu, 05 Mar 2026 09:06:19 +0000

If you've spent time in data engineering or analytics, you've almost certainly run into Parquet files. They show up everywhere — in data lakes, machine learning pipelines, cloud storage buckets, and more.

But what makes Parquet special, and how do you actually work with it in Python?

In this tutorial, I'll walk you through reading, writing, filtering, and compressing Parquet files using Python. By the end, you'll have a solid foundation for using Parquet in your own data pipelines.

You can get the code on GitHub.

What Is the Parquet File Format?
Prerequisites
Writing Parquet Files in Python
Reading Parquet Files in Python
Reading Specific Columns
Writing Parquet Files with Compression
Working with Row Groups and Filtering
A Practical Example: Analyzing Sales Data
When Should You Use Parquet?

What Is the Parquet File Format?

Parquet is an open-source columnar storage format originally developed by Cloudera and Twitter, and now part of the Apache ecosystem.

Unlike CSV or JSON, which store data row by row, Parquet stores data column by column. That might seem like a small implementation detail, but it has a big impact on performance.

Here's why Parquet is widely used:

Efficient compression — because all values in a column share the same data type, they compress far better than mixed-type rows
Column pruning — you can read only the columns you need, skipping the rest entirely
Schema preservation — data types are stored in the file, so integers don't become strings on read (and other subtle bugs)
Broad ecosystem support — works natively with Spark, Pandas, DuckDB, BigQuery, Athena, and more

Parquet is often compared to ORC (or optimized row columnar). Both are columnar formats, but Parquet has wider cross-platform support and tends to be the default choice outside of Hadoop/Hive environments.

Prerequisites

Before we get started, make sure you have:

Python 3.10 or a later version installed
Basic familiarity with pandas DataFrames
Some understanding of file I/O in Python

You'll need to install two libraries:

pip install pyarrow pandas

We'll use PyArrow as the engine for reading and writing Parquet files. You can also use [fastparquet](https://fastparquet.readthedocs.io/en/latest/) as an alternative engine, but PyArrow is the more commonly recommended choice.

Writing Parquet Files in Python

Let's start by creating a Parquet file. We'll build a small product inventory dataset and write it to disk.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Create a product inventory dataset
data = {
    'product_id': [1001, 1002, 1003, 1004, 1005],
    'product_name': ['Wireless Mouse', 'Mechanical Keyboard', 'USB-C Hub', 'Monitor Stand', 'Webcam HD'],
    'category': ['Peripherals', 'Peripherals', 'Accessories', 'Accessories', 'Peripherals'],
    'price': [29.99, 89.99, 49.99, 34.99, 74.99],
    'stock': [150, 80, 200, 95, 60]
}

df = pd.DataFrame(data)

# Convert to PyArrow Table and write as Parquet
table = pa.Table.from_pandas(df)
pq.write_table(table, 'inventory.parquet')

print("Parquet file created successfully!")

Output:

Parquet file created successfully!

Here's what's happening step by step. We start with a standard pandas DataFrame. Then we convert it to a PyArrow Table using pa.Table.from_pandas(). This is PyArrow's in-memory columnar representation. Finally, pq.write_table() serializes it to disk in Parquet format.

The conversion step is necessary because Parquet is a columnar format and PyArrow handles the translation between pandas' row-oriented structure and Parquet's column-oriented storage.

Reading Parquet Files in Python

Now let's read the file back:

# Read the Parquet file
table = pq.read_table('inventory.parquet')

# Convert to pandas for easy viewing
df_read = table.to_pandas()

print(df_read)
print(f"\nData types:\n{df_read.dtypes}")

Output:

   product_id         product_name     category  price  stock
0        1001       Wireless Mouse  Peripherals  29.99    150
1        1002  Mechanical Keyboard  Peripherals  89.99     80
2        1003            USB-C Hub  Accessories  49.99    200
3        1004        Monitor Stand  Accessories  34.99     95
4        1005            Webcam HD  Peripherals  74.99     60

Data types:
product_id        int64
product_name     object
category         object
price           float64
stock             int64
dtype: object

pq.read_table() loads the Parquet file into a PyArrow Table. We then call .to_pandas() to convert it into a familiar DataFrame.

Notice that data types are preserved correctly — price is float64, not a string. This is one of Parquet's practical advantages over CSV, where you often have to explicitly cast types after reading.

Reading Specific Columns

One of Parquet's most powerful features is the ability to read only the columns you actually need. This is called column pruning, and it can substantially reduce memory usage and load time on large files.

# Read only product name and price
table_subset = pq.read_table('inventory.parquet', columns=['product_name', 'price'])
df_subset = table_subset.to_pandas()

print(df_subset)

Output:

          product_name  price
0       Wireless Mouse  29.99
1  Mechanical Keyboard  89.99
2            USB-C Hub  49.99
3        Monitor Stand  34.99
4            Webcam HD  74.99

By passing a columns list to read_table(), PyArrow only reads those two columns from disk — the rest are skipped entirely. If your file has 100 columns but your query only needs 4, you're doing roughly 4% of the I/O work. That adds up fast at scale.

Writing Parquet Files with Compression

Parquet supports several compression codecs. Let's compare them on a larger dataset:

import os

# Generate a larger dataset
large_data = {
    'transaction_id': range(50000),
    'store': ['Downtown', 'Uptown', 'Suburban', 'Airport', 'Mall'] * 10000,
    'product': [f'product_{i % 200}' for i in range(50000)],
    'amount': [round(10 + (i % 500) * 0.5, 2) for i in range(50000)]
}

df_large = pd.DataFrame(large_data)
table_large = pa.Table.from_pandas(df_large)

# Write with different compression codecs
for codec in ['NONE', 'SNAPPY', 'GZIP', 'ZSTD']:
    path = f'transactions_{codec.lower()}.parquet'
    pq.write_table(table_large, path, compression=codec)
    size = os.path.getsize(path)
    print(f"{codec:<8}: {size:>10,} bytes")

Output:

NONE    :    635,565 bytes
SNAPPY  :    316,033 bytes
GZIP    :    177,654 bytes
ZSTD    :    158,082 bytes

Each codec makes a different trade-off between file size and speed:

NONE — no compression, fastest write, largest file
SNAPPY — fast compression and decompression, moderate size reduction, good for hot data
GZIP — better compression than Snappy, slower, good for archival
ZSTD — best compression ratio with competitive speed, generally the recommended default

For most production use cases, ZSTD is a solid default. It gives you the smallest files without a significant speed penalty.

Working with Row Groups and Filtering

Parquet files are internally divided into row groups — chunks of rows stored together. This structure enables a powerful optimization called predicate pushdown, where Parquet skips entire row groups that can't possibly match your filter condition.

Let's write a file with explicit row group sizing and then filter it:

# Write with smaller row groups (useful for filtering large files)
pq.write_table(
    table_large,
    'transactions_grouped.parquet',
    compression='ZSTD',
    row_group_size=10000
)

# Use PyArrow's dataset API for efficient filtering
import pyarrow.dataset as ds

dataset = ds.dataset('transactions_grouped.parquet', format='parquet')

# Filter: only transactions from the Downtown store over $150
filtered = dataset.to_table(
    filter=(ds.field('store') == 'Downtown') & (ds.field('amount') > 150)
)

df_filtered = filtered.to_pandas()
print(f"Matching transactions: {len(df_filtered)}")
print(df_filtered.head())

Output:

Matching transactions: 4300
   transaction_id     store      product  amount
0             285  Downtown   product_85   152.5
1             290  Downtown   product_90   155.0
2             295  Downtown   product_95   157.5
3             300  Downtown  product_100   160.0
4             305  Downtown  product_105   162.5

The ds.dataset() API with a filter argument doesn't load the entire file into memory. Instead, it evaluates the filter against each row group's metadata and skips groups that can't contain matching rows. The actual rows are only read when you call .to_table(). This is particularly useful for large files.

A Practical Example: Analyzing Sales Data

Let's put it all together with a better example. Say you receive daily sales exports and need to store them efficiently and query them by region and date.

from datetime import datetime, timedelta
import random

# Simulate 30 days of sales data
regions = ['North', 'South', 'East', 'West']
products = ['Laptop', 'Tablet', 'Headphones', 'Charger', 'Case']

records = []
start = datetime(2025, 1, 1)

for day in range(30):
    date = start + timedelta(days=day)
    for _ in range(200):
        records.append({
            'date': date.strftime('%Y-%m-%d'),
            'region': random.choice(regions),
            'product': random.choice(products),
            'units_sold': random.randint(1, 20),
            'revenue': round(random.uniform(50, 2000), 2)
        })

df_sales = pd.DataFrame(records)
table_sales = pa.Table.from_pandas(df_sales)

# Save as compressed Parquet
pq.write_table(table_sales, 'sales_jan2025.parquet', compression='ZSTD')
print(f"Saved {len(df_sales):,} records to Parquet")

# Query: total revenue by region for the first two weeks
dataset = ds.dataset('sales_jan2025.parquet', format='parquet')

first_two_weeks = dataset.to_table(
    columns=['region', 'revenue', 'date'],
    filter=ds.field('date') <= '2025-01-14'
).to_pandas()

summary = first_two_weeks.groupby('region')['revenue'].sum().sort_values(ascending=False)
print(f"\nRevenue by region (Jan 1–14):")
print(summary.round(2))

Output:

Saved 6,000 records to Parquet

Revenue by region (Jan 1–14):
region
West     752919.48
North    739783.22
South    730002.82
East     692128.58
Name: revenue, dtype: float64

This is a pattern you'll see in real data engineering work. Parquet handles the storage efficiently, column pruning limits what gets loaded into memory, and predicate pushdown ensures only the relevant date range is scanned.

When Should You Use Parquet?

Use Parquet when you:

Work with analytical or reporting workloads where you query specific columns
Need efficient, compressed storage for medium-to-large datasets
Store data in cloud platforms (S3, GCS, Azure Blob) that are billed by bytes scanned
Use tools like Spark, DuckDB, Athena, or BigQuery that have native Parquet support
Want schema enforcement across your pipeline

Don't use Parquet when you:

Need human-readable files; use CSV or JSON for that
Are doing frequent row-level updates or inserts; Parquet is write-once, read-many
Work with very small datasets where the overhead isn't worth it
Need streaming row-by-row writes; consider Avro instead

Conclusion

I hope you found this tutorial useful. Parquet's columnar layout, built-in compression, and schema preservation add up to faster queries, smaller files, and fewer type-related bugs.

In this tutorial, you learned how to write and read Parquet files, use column pruning to reduce I/O, apply compression codecs, and filter large files efficiently using predicate pushdown. These are the building blocks you'll reach for repeatedly in any data-heavy Python project.

Try swapping your next CSV export for Parquet and see how it affects your pipeline. Happy coding!

5 DIY Python Decorators for Building Cleaner Data Pipelines

Bala Priya C — Mon, 23 Feb 2026 07:30:20 +0000

Data pipelines often tend to accumulate the same boilerplate in a lot of places. You'll likely have retry logic copy-pasted across functions, timing code scattered through the codebase, validation checks buried inside business logic, and more. Decorators are a clean way to pull that stuff out.

They let you separate what a function does from how it behaves, and in data engineering, that's a useful separation to make.

This article walks through five practical decorators you can use to keep your data pipelines maintainable.

You can get the code on GitHub.

Prerequisites

Before we get started, you should be comfortable with:

Python functions as first-class objects — functions can be passed as arguments and returned from other functions
*args and **kwargs — used throughout to make decorators flexible enough to wrap any function signature
Basic exception handling — try/except/raise blocks appear in several decorators
functools module — specifically functools.wraps, explained in the next section

You don't need to have written a decorator for data pipelines before. That's what this article is for.

How Python Decorators Work

A decorator is a function that takes another function as input and returns a new function as output. The @ symbol is syntactic sugar to make it read cleanly.

Here's the simplest possible example:

def shout(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return str(result).upper()
    return wrapper

@shout
def greet(name):
    return f"hello, {name}"

print(greet("world"))
# HELLO, WORLD

When Python sees @shout above greet, it does this behind the scenes:

greet = shout(greet)

So calling greet("world") calls wrapper("world"), which calls the original greet, gets "hello, world", and returns "HELLO, WORLD". The original function is unchanged; it's just been wrapped.

The wrapper uses *args and **kwargs so it can forward any arguments without knowing them in advance. This is what makes decorators reusable across functions with different signatures.

Note: without @functools.wraps(func), your decorated function loses its original name and docstring. greet.__name__ would show wrapper instead of greet, which makes debugging and log output harder. Every decorator in this article includes functools.wraps for this reason.

1. `@retry` — Handle Transient Failures

In many projects, database connections and API calls often fail, a file is briefly locked by another process, and more. The common fix is wrapping every external call in a for loop with a try/except. But copy-pasting that pattern across your codebase gets messy fast.

Here's a reusable @retry decorator:

import time
import functools

def retry(max_attempts=3, delay=2, exceptions=(Exception,)):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    if attempt == max_attempts:
                        raise
                    print(f"Attempt {attempt} failed: {e}. Retrying in {delay}s...")
                    time.sleep(delay)
        return wrapper
    return decorator

Here we simulate a flaky SQLite read that fails the first two times before succeeding on the third attempt:

import sqlite3

# Setup: create an in-memory DB with some orders
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (order_id, customer_id, amount, status)")
conn.executemany("INSERT INTO orders VALUES (?, ?, ?, ?)", [
    (1, 101, 59.99,  "completed"),
    (2, 102, 120.00, "completed"),
    (3, 101, 34.50,  "refunded"),
    (4, 103, 89.00,  "completed"),
])
conn.commit()

# Simulate a connection that fails twice before succeeding
attempt_count = 0

@retry(max_attempts=3, delay=1, exceptions=(sqlite3.OperationalError,))
def fetch_completed_orders(conn):
    global attempt_count
    attempt_count += 1
    if attempt_count < 3:
        raise sqlite3.OperationalError("database is locked")
    cursor = conn.execute("SELECT * FROM orders WHERE status = 'completed'")
    return cursor.fetchall()


rows = fetch_completed_orders(conn)

print(rows)

Output:

Attempt 1 failed: database is locked. Retrying in 1s...
Attempt 2 failed: database is locked. Retrying in 1s...
[(1, 101, 59.99, 'completed'), (2, 102, 120.0, 'completed'), (4, 103, 89.0, 'completed')]

We're specific about sqlite3.OperationalError that covers locks and connection issues. A sqlite3.ProgrammingError from a malformed query is not worth retrying; it'll fail the same way every time.

There are three nested functions here because retry takes configuration arguments. retry(...) returns decorator, which is the actual decorator. decorator receives the function and returns wrapper.

2. `@timer` — Measure Execution Time

When a pipeline run is slower than expected, the naive approach is sprinkling time.time() calls through your code. It works, but timing logic ends up tangled with business logic everywhere, and you have to clean it all up when you're done. Let's write a @timer decorator:

import time
import functools

def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"[TIMER] {func.__name__} completed in {elapsed:.4f}s")
        return result
    return wrapper

Here we apply it to a transformation step that computes customer lifetime value and assigns spend tiers across a list of orders:

orders = [
    {"order_id": 1, "customer_id": 101, "amount": 59.99,  "status": "completed"},
    {"order_id": 2, "customer_id": 102, "amount": 120.00, "status": "completed"},
    {"order_id": 3, "customer_id": 101, "amount": 34.50,  "status": "refunded"},
    {"order_id": 4, "customer_id": 103, "amount": 89.00,  "status": "completed"},
    {"order_id": 5, "customer_id": 102, "amount": 45.00,  "status": "completed"},
]

@timer
def transform_orders(orders: list[dict]) -> list[dict]:
    # Compute lifetime value per customer
    ltv = {}
    for o in orders:
        if o["status"] == "completed":
            ltv[o["customer_id"]] = ltv.get(o["customer_id"], 0) + o["amount"]

    # Enrich each order with LTV and spend tier
    def spend_tier(value):
        if value >= 200: return "high"
        if value >= 100: return "mid"
        return "low"

    return [
        {**o, "customer_ltv": ltv.get(o["customer_id"], 0),
              "spend_tier": spend_tier(ltv.get(o["customer_id"], 0))}
        for o in orders
    ]


result = transform_orders(orders)

for row in result:
    print(row)

Output:

[TIMER] transform_orders completed in 0.0001s
{'order_id': 1, 'customer_id': 101, 'amount': 59.99, 'status': 'completed', 'customer_ltv': 59.99, 'spend_tier': 'low'}
{'order_id': 2, 'customer_id': 102, 'amount': 120.0, 'status': 'completed', 'customer_ltv': 165.0, 'spend_tier': 'mid'}
{'order_id': 3, 'customer_id': 101, 'amount': 34.5, 'status': 'refunded', 'customer_ltv': 59.99, 'spend_tier': 'low'}
{'order_id': 4, 'customer_id': 103, 'amount': 89.0, 'status': 'completed', 'customer_ltv': 89.0, 'spend_tier': 'low'}
{'order_id': 5, 'customer_id': 102, 'amount': 45.0, 'status': 'completed', 'customer_ltv': 165.0, 'spend_tier': 'mid'}

Inside wrapper, we store the result before printing the time and return it after, so the decorator is invisible to whatever called the function. We use time.perf_counter() instead of time.time() because it has higher resolution and isn't affected by system clock adjustments.

3. `@validate_schema` — Catch Bad Data Early

Data from upstream sources isn't always consistently shaped. A bug in a producer might start omitting a field. A schema change might rename a key. If a required field is missing, you want to know at the point the record enters your pipeline — not several transformations later when something raises a KeyError with no indication of which record caused it.

import functools

def validate_schema(required_keys):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(data, *args, **kwargs):
            if not isinstance(data, dict):
                raise TypeError(f"Expected dict, got {type(data).__name__}")
            missing = required_keys - data.keys()
            if missing:
                raise ValueError(f"Missing required keys: {missing}")
            return func(data, *args, **kwargs)
        return wrapper
    return decorator

We apply it to normalize_order, which standardizes each raw order record before it's passed to the transformation step:

@validate_schema(required_keys={"order_id", "customer_id", "amount", "status"})
def normalize_order(order: dict) -> dict:
    return {
        "order_id": order["order_id"],
        "customer_id": order["customer_id"],
        "amount": round(float(order["amount"]), 2),
        "status": order["status"].lower().strip(),
        "note": order.get("note", ""),   # optional — fine to be absent
    }


# Valid record — passes through
normalize_order({"order_id": 1, "customer_id": 101, "amount": "59.99", "status": "Completed"})
# {"order_id": 1, "customer_id": 101, "amount": 59.99, "status": "completed", "note": ""}

# Missing amount — caught immediately
normalize_order({"order_id": 2, "customer_id": 102, "status": "Completed"})
# ValueError: Missing required keys: {'amount'}

# Wrong type entirely — also caught
normalize_order([1, 101, 59.99, "completed"])
# TypeError: Expected dict, got list

Two checks run before the function body executes. First, we confirm the input is a dict. Second, we use set subtraction to find any missing required keys and raise a ValueError that names them exactly.

Note that note is not in required_keys. It's optional and we use .get() for it. The decorator only enforces what's truly required.

4. `@cache_result` — Skip Redundant Computation

Some pipeline steps are expensive and return the same result for the same inputs. A lookup table loaded from a database, a CSV read from disk, a slow aggregation — if you're calling these multiple times with the same arguments, there's no reason to recompute it.

Python has functools.lru_cache built in, but it only works with hashable arguments. Dicts and lists — common in data pipelines — won't work with it. This decorator handles those cases by hashing the arguments:

import functools
import hashlib
import json

def cache_result(func):
    cache = {}

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        key = hashlib.md5(
            json.dumps((args, kwargs), sort_keys=True, default=str).encode()
        ).hexdigest()

        if key not in cache:
            cache[key] = func(*args, **kwargs)
            print(f"[CACHE] Computed result for {func.__name__}")
        else:
            print(f"[CACHE] Cache hit for {func.__name__}")

        return cache[key]

    wrapper.cache_clear = lambda: cache.clear()
    return wrapper

We apply it to load_discount_codes, which reads a reference table from our SQLite database. This table is queried repeatedly during enrichment but rarely changes within a run:

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE discount_codes (code TEXT, discount_pct REAL, active INTEGER)")
conn.executemany("INSERT INTO discount_codes VALUES (?, ?, ?)", [
    ("SAVE10", 10.0, 1),
    ("SAVE20", 20.0, 1),
    ("OLD5",    5.0, 0),
])
conn.commit()

@cache_result
def load_discount_codes(active_only: bool = True) -> dict:
    query = "SELECT code, discount_pct FROM discount_codes"
    if active_only:
        query += " WHERE active = 1"
    rows = conn.execute(query).fetchall()
    return {code: pct for code, pct in rows}


# First call hits the database
codes = load_discount_codes(active_only=True)

# Second call is instant
codes = load_discount_codes(active_only=True)

print(codes)

# Different argument = separate cache entry
all_codes = load_discount_codes(active_only=False)

Output:

[CACHE] Computed result for load_discount_codes
[CACHE] Cache hit for load_discount_codes
{'SAVE10': 10.0, 'SAVE20': 20.0}
[CACHE] Computed result for load_discount_codes

The cache key is generated by serializing args and kwargs to JSON, then MD5-hashing the result. sort_keys=True ensures consistent output regardless of keyword argument order; default=str handles types that JSON can't serialize natively. The cache dict lives in the closure and persists for the program's lifetime. cache_clear is attached to the wrapper as an escape hatch if you need to force a fresh load.

5. `@log_step` — Build a Trail

Without structured logging, debugging a failed pipeline run means guesswork. You have no record of what ran, what succeeded, or where things went sideways.

import functools
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s — %(message)s")
logger = logging.getLogger(__name__)

def log_step(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        logger.info(f"START  {func.__name__}")
        try:
            result = func(*args, **kwargs)
            logger.info(f"END    {func.__name__} — OK")
            return result
        except Exception as e:
            logger.error(f"FAIL   {func.__name__} — {type(e).__name__}: {e}")
            raise
    return wrapper

We apply it across the main stages of our pipeline:

@log_step
def fetch_completed_orders(conn) -> list[tuple]:
    cursor = conn.execute("SELECT * FROM orders WHERE status = 'completed'")
    return cursor.fetchall()

@log_step
def transform_orders(orders: list[dict]) -> list[dict]:
    ...

@log_step
def write_results(conn, rows: list[dict]) -> int:
    conn.execute("CREATE TABLE IF NOT EXISTS orders_transformed "
                 "(order_id, customer_id, amount, status, customer_ltv, spend_tier)")
    conn.executemany(
        "INSERT INTO orders_transformed VALUES (:order_id, :customer_id, :amount, "
        ":status, :customer_ltv, :spend_tier)",
        rows,
    )
    conn.commit()
    return len(rows)

After logging the error, we call bare raise. This re-raises the original exception so the pipeline still fails. Without it, the decorator would silently swallow every error and execution would continue. Always re-raise in a logging decorator.

Wrapping Up

Decorators let you write that logic once and apply it anywhere. The five patterns here — retry, timer, schema validation, caching, and logging — cover the most common cross-cutting concerns in data pipelines.

A good starting point is @log_step and @retry. They address the most immediate pain points in data pipelines with minimal setup. Add the others as your pipeline grows and the needs become clearer.

Happy coding!

MIT's Missing Semester Class: Beyond the CS Curriculum

Bala Priya C — Fri, 04 Aug 2023 09:51:04 +0000

I was in my second semester of grad school when I learned about MIT’s missing semester class. Over the next few weeks, I worked through the class and found it super helpful. Here's why.

In a conventional CS curriculum, you’ll diligently work through, perhaps, two semester-long classes, perfecting your data structures and algorithms foundations (yeah, also count the weekends of leetcoding). You'll also take classes in compiler design, operating systems, and much more. These are core computer science skills.

But what you don't learn at the university, though, is how to exit Vim (yeah, I know!) or how to git rebase with confidence (add as many as you’d like).

Rightly called The Missing Semester (of Your CS Education), this class from MIT will teach you how to use some of the tools that are fundamental to the software engineering ecosystem. From shell scripting to the fundamentals of information security—spanning around 12 lectures—you can add a bunch of practical skills to your toolbox.

And if you’re into CS, chances are you’ve already heard of or taken this class. If you’ve taken it before, I’m sure you enjoyed the process. But if you haven’t already, happy learning!

So What Does This Course Teach?

Here’s what I did when I took the class (very studently of me):

Watched the lectures on YouTube; took notes and coded along where possible.
Read through the lecture notes on missing.csail.mit.edu.
Completed the exercises for each module.

I’ll briefly go over the topics covered in the lectures/modules.

#1 – Shell and Shell Scripting

From automating ~~boring~~ simple tasks to building ETL pipelines in production, shell scripting is the swiss-army knife of a programmer’s toolbox.

The first lecture is an introduction to shell (and the class). And the second lecture covers shell scripting fundamentals. Be sure to do the exercises.

#2 – Working with Vim

Yes, I had to spend a couple of dedicated days learning my way around Vim. The lecture on Vim covers basics like navigating and editing files. It also covers customizing Vim and advanced features like macros.

#3 – Data Wrangling

To me, this was, perhaps, the most challenging module. Learning about regular expressions and tools like sed and awk—all in a single lecture—was quite overwhelming.

#4 – Command-Line Environment

This lecture introduces the learners to command-line capabilities including process management, all things ssh for working with remote machines, dotfiles, and more.

#5 – Version Control with Git

Even if you are a student, you should learn to use Git. It can be helpful in working on your projects as well as in your internships.

The lecture emphasizes how we can do better than just knowing our way around a few Git commands, and covers Git’s data model. And goes on to cover the usage of basic to advanced commands.

#6 – Debugging and Profiling

Debugging and profiling is one of my favorite modules from the class. From native (or should rather say naïve) print statements to using the shell program logger and Python debugger, this covers reasonable depth. You’ll also learn tracing system calls with strace.

In the profiling part, you will learn profiling for CPU usage, memory, and visualizing profiles to make sense of them better. In addition, you’ll get to know using perf for event profiling and tools for resource monitoring.

#7 – Build Systems, Dependency Management, and More

Originally titled as metaprogramming, this lecture provides a comprehensive overview of build systems, dependency management, continuous integration systems, and testing.

#8 – Security and Cryptography

This lecture is an introduction to hash functions and cryptography basics. One of the easiest lectures where you don't feel overwhelmed at all.

#9 – Potpourri

True to its name, this lecture is a collection of interesting hacks. I particularly liked the overview of daemon processes and keyboard remapping.

Wrapping Up

So if you are a student or a new grad, the tools you learn in this class should be helpful in your software engineering career. Even if you are looking to pivot to software engineering from another domain, you can work through this class after you’ve gained some programming experience.

So have you watched the first lecture yet?

How To Code a Simple Number Guessing Game in Python

Bala Priya C — Thu, 27 Jul 2023 02:30:28 +0000

I spent the last weekend compiling a list of games you can code in Python. But why?

If you're a beginner Python programmer, building fun games will help you learn the language faster—and better—without getting bogged down in the syntax and the like. I built some of these games when I was learning Python; I quite enjoyed the process!

The first game you can code—and the simplest of them all—is a number guessing game (or Guess the Number!). So I thought I'd write a step-by-step tutorial to code this game—and help beginners learn some of the fundamentals along the way.

Let's begin!

How Does the Number Guessing Game Work?

In a number guessing game, the user guesses a randomly generated secret number within a given number of attempts.

After each guess, the user gets hints on whether their guess is too high, too low, or correct. So yeah, the game ends when the user guesses the secret number or runs out of attempts.

Coding the Number Guessing Game

Let's get to coding! Create a new Python script and code along.

Step 1 - Import the `random` module

Let's start by importing the built-in random module. The random module has functions we can use to generate a random secret number within the specified range:

import random

Note: The random module gives pseudo-random numbers—and not truly random numbers. So don't use it for sensitive applications such as password generation.

Step 2 - Set up the range and the maximum number of attempts

Next, we need to decide on the range for the secret number and the maximum number of attempts allowed for the player. For this tutorial, let's set the lower_bound and upper_bound to 1 and 1000, respectively. Also, set the maximum attempts allowed max_attempts to 10:

lower_bound = 1
upper_bound = 1000
max_attempts = 10

Step 3 - Generate a random number

Now, let's generate a random number within the specified range using the random.randint() function. This is the secret number that the user needs to guess:

secret_number = random.randint(lower_bound, upper_bound)

Step 4 - Read in the user's input

To get input from the user, let's create a function called get_guess(). Remember, the user can enter an invalid input: a number outside the range [lower_bound, upper_bound], a string or a floating point number, and more.

We handle this in the get_guess() function that continuously prompts the user to enter a number—within the specified range—until they provide a valid input.

Here, we use a while loop to prompt the user for a valid input until they enter an integer between lower_bound and upper_bound:

def get_guess():
    while True:
        try:
            guess = int(input(f"Guess a number between {lower_bound} and {upper_bound}: "))
            if lower_bound <= guess <= upper_bound:
                return guess
            else:
                print("Invalid input. Please enter a number within the specified range.")
        except ValueError:
            print("Invalid input. Please enter a valid number.")

Step 5 - Validate the user's guess

Next, let's define a check_guess() function that takes the user's guess and the secret number as inputs and provides feedback on whether the guess is correct, too high, or too low.

The function compares the player's guess with the secret number and returns a corresponding message:

def check_guess(guess, secret_number):
    if guess == secret_number:
        return "Correct"
    elif guess < secret_number:
        return "Too low"
    else:
        return "Too high"

Step 6 - Track the number of attempts and detect end-of-game conditions

We'll now create the function play_game() that handles the game logic and puts everything together. The function uses the attempts variable to keep track of the number of attempts made by the user. Within a while loop, the user is prompted to enter a guess that's processed by the get_guess() function.

The call to the check_guess() function provides feedback on the user's guess:

If the guess is correct, the user wins, and the game ends.
Otherwise, the user is given another chance to guess.
And this continues until the player guesses the secret number or runs out of attempts.

Here's the play_game() function:

def play_game():
    attempts = 0
    won = False

    while attempts < max_attempts:
        attempts += 1
        guess = get_guess()
        result = check_guess(guess, secret_number)

        if result == "Correct":
            print(f"Congratulations! You guessed the secret number {secret_number} in {attempts} attempts.")
            won = True
            break
        else:
            print(f"{result}. Try again!")

    if not won:
        print(f"Sorry, you ran out of attempts! The secret number is {secret_number}.")

Step 7 - Play the game!

Finally, you can call the play_game() function every time the Python script is run:

if __name__ == "__main__":
    print("Welcome to the Number Guessing Game!")
    play_game()

Putting it all together

Now our Python script looks like so:

# main.py
import random

# define range and max_attempts
lower_bound = 1
upper_bound = 1000
max_attempts = 10

# generate the secret number
secret_number = random.randint(lower_bound, upper_bound)

# Get the user's guess
def get_guess():
    while True:
        try:
            guess = int(input(f"Guess a number between {lower_bound} and {upper_bound}: "))
            if lower_bound <= guess <= upper_bound:
                return guess
            else:
                print("Invalid input. Please enter a number within the specified range.")
        except ValueError:
            print("Invalid input. Please enter a valid number.")

# Validate guess
def check_guess(guess, secret_number):
    if guess == secret_number:
        return "Correct"
    elif guess < secret_number:
        return "Too low"
    else:
        return "Too high"

# track the number of attempts, detect if the game is over
def play_game():
    attempts = 0
    won = False

    while attempts < max_attempts:
        attempts += 1
        guess = get_guess()
        result = check_guess(guess, secret_number)

        if result == "Correct":
            print(f"Congratulations! You guessed the secret number {secret_number} in {attempts} attempts.")
            won = True
            break
        else:
            print(f"{result}. Try again!")

    if not won:
        print(f"Sorry, you ran out of attempts! The secret number is {secret_number}.")

if __name__ == "__main__":
    print("Welcome to the Number Guessing Game!")
    play_game()

Here's the output from a sample run of the script:

Welcome to the Number Guessing Game!
Guess a number between 1 and 1000: 500
Too low. Try again!
Guess a number between 1 and 1000: 750
Too high. Try again!
Guess a number between 1 and 1000: 625
Too low. Try again!
Guess a number between 1 and 1000: 685
Too low. Try again!
Guess a number between 1 and 1000: 710
Too low. Try again!
Guess a number between 1 and 1000: 730
Congratulations! You guessed the secret number 730 in 6 attempts.

Wrapping Up

Congratulations! You've successfully built a number guessing game in Python. I'll see you all soon in another tutorial. But don't wait for me. Check out other games you can build—features you can code—and start coding!

How To Write an Effective Technical Résumé

Bala Priya C — Tue, 04 Jan 2022 15:41:19 +0000

Getting your résumé writing right is an important step in your developer journey.

Have you ever been a part of the job search process—be it an internship or a full-time opportunity—as a student or a new grad?

If yes, you already know: getting past the résumé screening step, and landing that interview call can often be harder than the actual interview.

Over the next few minutes, you'll get to know some actionable tips for résumé writing, that you could use to revamp your résumé. This post is inspired by Jessie Newman's webinar for WWCode, NYC chapter.

Let's get started.

What goes on a résumé?

Let's start with this question:

What are companies and hiring managers looking for?

Well, they're looking for candidates who:

can improve the company's products with their technical expertise,
be enjoyable to work with, and
can contribute positively to the company's culture and growth.

Even if the recruiter skims through your résumé for less than a minute, you should stand out as a prospective candidate, yes?

For this to happen, the content on your résumé should be:

Recent: Always present information in reverse chronological order—starting with the most recent experience first.
Relevant to the role that you're applying for.
Clear even to a reader who has no context.

Typically, your résumé should only be about a page long. And that's all you've got to make an impression on the reviewer.

As they say, "You're much more than a one-page résumé—but your résumé should not be more than one page.😄

Format of a résumé

There's no one recommended format to draft your résumé. However, the following sections should typically be present:

- Name and Contact info
- Objective (optional)
- Education
- Technical Experience
- Skills
- Leadership | Volunteering

Let's now visit each of these sections, and see how you can best structure each of them.

Name and contact info

✅ Include your name, your email address, links to your portfolio/GitHub.

✅ Be sure to check if your email address is professional enough.

✅ Include social media handles—like LinkedIn—only if you've updated them.

Objective (`optional`)

Include the objective section only if you aim at providing some context to the reviewer.

Every line should tell the recruiter something that they don't already know.

For example, if you're a CS major applying to a software engineering role, your objective isn't going to provide any context to the recruiter.

On the other hand, suppose you're a professional accountant, who's looking to break into software development. Then, the objective tells the recruiter upfront that you're trying to switch careers—and they won't look for a CS degree or developer experience as they skim through your résumé.

Education

Always cite details of your education—starting from your highest qualification first.

If you're a Master's student, only mention details of your Master's and undergraduate degree. Some companies do have a certain cut-off GPA, so be sure to include your GPA.

Some people do include Relevant Coursework subsection in their Education tab.

However, you should use it only if needed.

Being a CS major, doing courses in algorithm design and analysis, and operating systems isn't any interesting to the reviewer. If you're from a non-CS stream, but have supplemented your coursework with courses from the CS bucket—you may include them in the Relevant Coursework section.

Technical Experience

This section should account for nearly 80% of your résumé, and should include:

Relevant work experience and
Projects

▶ We'll talk about this in greater detail in the next section.

Skills

You should always organize your skills by category—ordered by proficiency.

Here's an example:

Languages: Python, JavaScript
Libraries: NumPy, pandas, scikit-learn
Tools: Git

You should always remember to demonstrate your skills in the other sections.

For example, if Python is the language that you're most proficient in—your projects should be indicative of your proficiency.

Leadership | Volunteering

If you've been volunteering, involved in open-source communities, mentoring and the like, you may include them in this section.

Ensure that you're communicating your interests and impact clearly, and keep this section short and towards the end of your résumé.

How to describe your technical experience

This section is the most crucial section in your résumé—be sure to draft this section carefully.

Here are a few suggestions on how you should explain your experience and projects.

❌ Do not list down your job responsibilities.
✔ Write what you accomplished.

❌ Do not tell what you learned.
✔ Instead, explain what you built with that knowledge.

❌ Do not use weak language.

Avoid phrases like:

- Helped build, 
- worked as part of the team,
- helped implement

✔ Use strong language that's impactful.

Say:
- Built,
- Worked on,
- Implemented

❌ Do not be vague when specifying impact.

...worked on speeding up the inference pipeline 
--> # not quantifying impact

✔ Quantify impact wherever possible—talk numbers!.

...worked on speeding up the inference pipeline by 30% 
by reducing the inference time to 2.5 ms --> #quantifying impact

❌ Do not include many projects without explaining each of them.
✔ Explain your projects clearly in detail—prioritize quality over quantity.

Be sure to specify the programming language, and tech stacks used.

Built a ____ using JavaScript, React.

Used Python to code a process scheduler.

Never leave the recruiter guessing why the project is interesting/relevant. Explain clearly.

Now that you know how to draft all major sections in your résumé, let's list down a few concluding points.

Points to remember

Have a résumé for every role that you'd be applying to.

If you're interested in both software engineering and data analytics, be sure to draft a dedicated résumé for each of these roles.

Don't be terse in explaining your projects—including portfolio links doesn't suffice.

The recruiter may not have time to look at your portfolio. So your résumé should do the talking for you.

Do not use fuzzy language just to make your projects sound cool and complicated.

Prefer using simple and clear language instead—just the way you'd explain to your friends.

🎯 And you've reached the end of this post on best practices in résumé writing.

Conclusion

Thanks for reading all the way up to here!😄

Hope you found this post helpful. If you know someone who'd find this useful, do share with them as well.

If you're currently looking for opportunities, I wish you loads of luck in your endeavors.

String Slicing in Python, Explained

Bala Priya C — Mon, 03 Jan 2022 09:35:33 +0000

String slicing lets you slice into Python strings, and work with their slices, or substrings—instead of the whole string.

As strings in Python are immutable, you cannot change them in place. In this tutorial, you'll learn to slice Python strings and work with substrings.

Python String Slicing Syntax

<str>[start:stop:step]

The above line of code:

Returns a slice of the string <str>—starting at index start, extending up to stop-1 in steps of step.
The start index is optional: the slice starts from the beginning of the string by default.
The stop index is also optional: the slice extends up to the end of the string by default.
The step value is optional too. The default value of step is 1 and includes all characters in the string.

Python String Slicing Example

my_str = "Python3"

▶ Let's use enumerate() and examine the characters at each index in the string.

for idx,char in enumerate(my_str):
  print(f"At index {idx}: letter {char}")

# Output
At index 0: letter P
At index 1: letter y
At index 2: letter t
At index 3: letter h
At index 4: letter o
At index 5: letter n
At index 6: letter 3

In the above code, you've used the enumerate() function in conjunction with the for loop. This lets you loop through iterables, and access items along with their indices simultaneously—without having to use the range() function to get the indices.

▶ Let's now use string slicing.

# With `start` and `stop` indices
print(my_str[1:6])
# Output: ython

# Without `stop` index
print(my_str[1:])
# Output: ython3

# Without `start` index
print(my_str[:5])
# Output: Pytho

# With `step = 2`, slice includes every second character
print(my_str[::2])
# Output: Pto3

# Without `start`, `stop` and `step`: slice is entire string
print(my_str[::])
# Output: Python3

Python String Slicing with Negative Step

When you set step to a negative value, you can get slices starting from the end of the string—reverse substrings.

If step = -1 you get a slice starting from the end of the string, and including every character.

This can be super handy when you'd like to reverse a string, like this:

print(my_str[::-1])

# Output: 3nohtyP

Conclusion

To sum up, <str>[start:stop:step] is the syntax to obtain string slices or substrings in Python.

Now that you've learned how to slice strings, it's time to put your skills to practice, maybe? Happy learning and coding!

Cover image: Photo by Tamanna Rumee on Unsplash

A 2021 Reflection Journal: My Tech Writing Journey, Learning, and More

Bala Priya C — Thu, 30 Dec 2021 10:49:40 +0000

With the COVID pandemic bringing about a paradigm shift to virtual, a lot of us got to explore and unleash new opportunities—despite all the odds!

I got into technical content writing roughly a year ago. And 2021 has helped me etch a good growth curve in writing.

Getting started with writing

In late 2020, I had started writing tech tutorials on Medium. All my tutorials were summaries of what I was learning back then. So writing those posts really helped me fill my gaps in understanding, and the feedback from the readers motivated me to keep going.

However, as Medium was a paid platform with articles on all things life, work, and beyond—I was looking to start writing on a free platform, where I'd also get to read more developer-focused content—and get inspired by experienced developers.

In November 2020, I joined the writing team at OpenMined, an open-source organization working in the realm of Privacy-Preserving AI. There, I started learning more about the privacy principles guiding ML and the like. I also started summarizing talks from their Privacy Conference in the form of blog posts for the community.

Thus ended 2020; To new beginnings! Hello 2021🎉✨

Leading the writing team at OpenMined

In January, I got the opportunity to lead the writing team at OpenMined. Leading a team of over 30 writers from over 10 time zones—was a challenging yet rewarding experience.

I led the writing team for a over 6 months, from January through June 2021. During this period, I coordinated efforts to ensure consistent content on the OpenMined blog that members of the community could learn from. And I continued to write a few posts as well.

If you're interested in reading my posts on privacy, and privacy-preserving AI, I've linked to my author profile here

Joining DEV community

I had also started writing on dev.to in early 2021. I liked the fact that I could now read a lot of well-written tutorials on a wide variety of tech stacks and loved how DEV community is run by the devs for the devs.

I spent my first few weeks reading more, and writing less.

I found features like creating a series, being able to include the canonical URL, writing in Markdown, and the fact that all of the content was accessible for free super cool. And I continued writing.😊

I was following writers like Chris Bongers @dailydevtips1 and could only get inspired more!

Writing for freeCodeCamp

I was trying to code my way through freeCodeCamp's Responsive Web Design curriculum, when I got to know that I can apply to become a writer for freeCodeCamp's publication. ✨ I was super happy when Abbey (Abigail Rennemeyer), who is the managing editor onboarded me as a contributor shortly after.

From June to November 2021, I wrote around 18 tutorials—predominantly covering topics in core Python.

You can find all posts that I've written for freeCodeCamp in my author profile. And according to stats from Google Analytics, readers spend 1000+ hours every month reading my beginner-friendly tutorials.

Joining AI Time Journal as an associate editor

I came across another interesting volunteering opportunity at the AI Time Journal while scrolling through my LinkedIn feed. It was regarding an associate editor role with their team. As an associate editor, I had the opportunity to interview guests—who were all thought leaders and experts in data science, and subsequently publish interview posts featuring them.

The guests that I got to interview include startup founders in the data-driven decision making and privacy space, and engineering and program managers at Google, Meta (then, Facebook 😄) and the like.

In case you're interested in data science, you can find all the interviews in this link.

A much-needed break

As a grad student who was and still is trying hard to get past the finish line, I had a burnout. And I decided to stay off the social media radar for a few weeks—deleting a couple of accounts permanently, and checking even school emails sparingly.

This much-needed break indeed helped me get back on track, and regain the lost momentum.

Ending 2021 on a high note

During one of the conversations, @atapas (Tapas Adhikary)—one of the best content creators we have—had suggested that I should also consider writing on Hashnode for there's a vibrant and supportive community around it. And I had made my decision at once.😄

Inspired by great and consistent writers like Tapas and Chris, I decided to join Hashnode. It has been a little over a week, and I love the community already.

And I certainly look forward to a great experience!

I'm super thankful to wonderful communities like OpenMined, freeCodeCamp, dev.to, and Hashnode for having me in 2021. This year has indeed been a great year!

The road ahead!

Though I'm happy with my progress, there's a lot of scope for improvement, and there's a lot I could have done better. But I'm going to be easy on me, and try to get them all right in the coming year. 😄

I'd like to code and write more consistently on Hashnode and dev.to, complete the 100DaysofCode challenge, and help others in their coding journey.

And many congratulations to each of you for staying courageous, and pulling through another difficult year. Do take a moment to reflect on your journey, and be thankful for how far you've come! ✨

For they say, "The more thankful you are for what you have—the more you'll have to be thankful for!"

Thanks again for reading, and have a great year ahead!🎉🎊

PyDP: A Python Differential Privacy Library

Bala Priya C — Mon, 27 Dec 2021 13:12:00 +0000

Outline

1️⃣ What does Differential Privacy try to address?
2️⃣ Why doesn't anonymization suffice?
3️⃣ PyDP example walkthrough

What does Differential Privacy try to address?

Differential privacy aims at addressing the paradox of learning nothing about an individual while learning useful information about a population. In essence, it describes the following promise, made by a data holder, or curator, to a data subject:

“You will not be affected, adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, data sets, or information sources, are available.” – Cynthia Dwork in The Algorithmic Foundations of Differential Privacy.

Differential Privacy ensures that any sequence of outputs, which are responses to queries is ‘essentially’ equally likely to occur, independent of the presence or absence of any individual’s record.

Consider the illustration below, where the two databases Database#1 and Database#2 differ by only one record, say, your data. If the results obtained from querying the database under these two different settings, are almost the same or similarly distributed, then they essentially are indistinguishable to an adversary.

Illustration of Differentially Private Database Mechanism (Image Source)

Mathematically,

P r [M (d) \in S] \leq e x p (ϵ) P r [M (d') \in S]

where, d and d’ are two subsets of data that differ by a single training example.

M(d) is the output of the training algorithm for the training subset d and M(d’) is the output of the training algorithm for the training subset d’.
The probabilities that these outputs belong to a specific set S under both these conditions should be arbitrarily close. The above equation should hold for all subsets d and d’.

Smaller the value of Ɛ, stronger the privacy guarantees.

Membership Inference Attack (MIA) attempts at determining the presence of a record in a machine learning model’s training data by querying the model. From the discussion above, as the inclusion or exclusion of an individual’s data record cannot be inferred, differential privacy ensures protection against such attacks.

Differentially private database mechanisms can therefore, make confidential data widely available for accurate data analysis.

Why doesn't anonymization suffice?

The Netflix Prize was an open competition for the best collaborative filtering algorithm for movie recommendations.

The dataset released was anonymized, without the users or the films being identified except by numbers assigned for the contest. Such anonymized movie records were published by Netflix as training data for the competition.

However, there were several users who could be identified by linkage with the Internet Movie Database (IMDb) which was non-anonymized and publicly available.

Researchers Arvind Narayanan and Vitaly Shmatikov, at the University of Texas at Austin present their studies in their work Robust De-anonymization of Large Datasets (How to Break Anonymity of the Netflix Prize Dataset).

Therefore, such linkage attacks can be used to match “anonymized” records with non-anonymized records in a different dataset.

Differential privacy aims at neutralizing such linkage attacks.
As Differential Privacy is a property of the data access mechanism, it is unrelated to the presence or absence of auxiliary information available to the adversary.

Therefore, access to the IMDb would no longer permit a linkage attack to someone whose history is in the Netflix training set than to someone not in the training set.

De-anonymization of users in the Netflix Prize contest (Image Credit: Arvind Narayanan)

PyDP Example Walkthrough

PyDP is OpenMined’s Python wrapper for Google’s Differential Privacy project. The library provides a set of ε-differentially private algorithms, which can be used to produce aggregate statistics over numeric datasets containing private or potentially sensitive information.

Installing PyDP

!pip install python-dp # installing PyDP

Necessary Imports

import pydp as dp # by convention our package is to be imported as dp (dp for Differential Privacy!)
from pydp.algorithms.laplacian import BoundedSum, BoundedMean, Count, Max
import pandas as pd
import statistics 
import numpy as np
import matplotlib.pyplot as plt

Fetch all the required data!

The dataset used here contains 5000 records, and is stored across 5 files, each file containing 1000 records.

More specifically, the dataset contains details such as the first and last names, email addresses of customers and the amount they spent on purchasing goods, and the state in the US they're from.

Let's fetch all the records, read them into pandas DataFrames and take a look at the head of each of the DataFrames.

url1 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/01.csv'
df1 = pd.read_csv(url1,sep=",", engine = "python")
print(df1.head())

url2 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/02.csv'
df2 = pd.read_csv(url2,sep=",", engine = "python")
print(df2.head())

url3 ='https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/03.csv'
df3 = pd.read_csv(url3,sep=",", engine = "python")
df3.head()

url4 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/04.csv'
df4 = pd.read_csv(url4,sep=",", engine = "python")
print(df4.head())

url5 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/05.csv'
df5 = pd.read_csv(url5,sep=",", engine = "python")
print(df5.head())

Now that we've fetched records from all the 5 files, let us concatenate all the DataFrames into a single large DataFrame and this constitutes our original dataset. Note that our dataset has 5000 rows(records) and 6 columns.

combined_df_temp = [df1, df2, df3, df4, df5]
original_dataset = pd.concat(combined_df_temp)
print(original_dataset.shape)

# Result
# (5000,6)

Creating a Parallel Database

Let us now create a parallel database that differs by only one record, say, Osbourne's record and name it redact_dataset. We then inspect the heads of both DataFrames to verify that Osbourne's record has been removed.

redact_dataset = original_dataset.copy()
redact_dataset = redact_dataset[1:]
print(original_dataset.head())
print(redact_dataset.head())

At this point, let us ask ourselves the following question.

Is the amount of money we spend at our neighborhood store private or sensitive information? Well, it may not seem all that sensitive! But, what if the same information can be used to identify us?

In the example that we have, let us say we remove all personal information such as name and email address. Given that there's some access to the store's sales record, will the sales amount in itself not suffice to infer Osbourne's identity? Yes!

And to do that, we sum up all entries in the sales_amount column in our original dataset, and the redact_dataset. The difference between these two sums exactly gives us the amount that Osbourne spent, and is verified as shown in the code snippet below.

This is a simple example where membership inference was successful even after removal of personally identifiable information.

sum_original_dataset = round(sum(original_dataset['sales_amount'].to_list()), 2)
sum_redact_dataset = round(sum(redact_dataset['sales_amount'].to_list()), 2)
sales_amount_Osbourne = round((sum_original_dataset - sum_redact_dataset), 2)
assert sales_amount_Osbourne == original_dataset.iloc[0, 4]

Differentially Private Sum

Now, we illustrate how differentially private sum in place of simple sum can help in rendering membership inference attacks unsuccessful.

For the example above, let's assume that the customers should spend a minimum of 5$ at the store and no more than 250$ for a particular purchase.

We then go ahead and compute differentially private sum on both original and the parallel dataset that differed by one record, as shown in the code snippets below.

dp_sum_original_dataset = BoundedSum(epsilon= 1.5, lower_bound =  5, upper_bound = 250, dtype ='float') 
dp_sum_og = dp_sum_original_dataset.quick_result(original_dataset['sales_amount'].to_list())
dp_sum_og = round(dp_sum_og, 2)
print(dp_sum_og)

# Output dp_sum_og
# 636723.61
dp_redact_dataset = BoundedSum(epsilon= 1.5, lower_bound =  5, upper_bound = 250, dtype ='float')
dp_redact_dataset.add_entries(redact_dataset['sales_amount'].to_list())
dp_sum_redact=round(dp_redact_dataset.result(), 2)
print(dp_sum_redact)

# Output dp_sum_redact
# 636659.17

Let's proceed to summarize a few observations.

Now that we've calculated the differentially private sum on the original and the second dataset, it's straightforward to verify that that the differentially private sums are not equal to sums under the non-differentially private setting.
Also, the difference is no longer equal to the amount that Osbourne spent indicating that membership attacks would now be unsuccessful, regardless of access to any other customer records.
Interestingly, the differentially private sum values are still comparable and are not very different.

We've therefore succeeded in ensuring differential privacy in our simple example!

print(f"Sum of sales_value in the orignal dataset: {sum_original_dataset}")
print(f"Sum of sales_value in the orignal dataset with DP: {dp_sum_og}")
assert dp_sum_og != sum_original_dataset

# Output
Sum of sales_value in the orignal dataset: 636594.59
Sum of sales_value in the orignal dataset with DP: 636723.61

print(f"Sum of sales_value in the second dataset: {sum_redact_dataset}")
print(f"Sum of sales_value in the second dataset with DP: {dp_sum_redact}")
assert dp_sum_redact != sum_redact_dataset

# Output
Sum of sales_value in the second dataset: 636562.65
Sum of sales_value in the second dataset with DP: 636659.17

print(f"Difference in Sum with DP: {round(dp_sum_og - dp_sum_redact, 2)}")
print(f"Actual Difference in Sum: {sales_amount_Osbourne}")
assert round(dp_sum_og - dp_sum_redact, 2) != sales_amount_Osbourne

# Output
Difference in sum using DP: 64.44
Actual Value: 31.94

Hope this introductory post helped in understanding the intuition behind differential privacy and protection against membership inference attacks. We shall look at a few more examples in subsequent blog posts.😊

References

[1] The Algorithmic Foundations of Differential Privacy by Cynthia Dwork.

[2] PyDP Tutorial by Chinmay Shah at OpenMined Privacy Conference

Private AI: Machine Learning on Encrypted Data

Bala Priya C — Fri, 24 Dec 2021 13:25:03 +0000

Kristin Lauter is currently the head of research science at Facebook AI Research (Meta), formerly the head of Cryptography and Privacy Research Group at Microsoft Research. This post is based on her talk at the OpenMined Privacy Conference.

What is the privacy problem with AI?

Let us begin by looking at a generic Machine Learning (ML) algorithm that takes in our data as input and outputs some kind of decision - a classification label, numerical value or a recommendation.

The privacy problem stems from the fact that we have to input our data to get those nice and valuable predictions.

Many AI applications powered by smart agents are hosted on the cloud, and protecting privacy of user data is pivotal to building secure applications.

The privacy of data can be protected through encryption. However, standard encryption methods do not allow computation on encrypted data. Here's where Homomorphic Encryption (HE) helps.

What is Homomorphic Encryption?

In simple terms, Homomorphic Encryption is a mathematical tool:

that allows for encryption of data,
ensuring privacy while at the same time, allowing computations to be performed on the encrypted data.
The result of computation can then be decrypted to get the results.

As shown in the figure below, with Homomorphic Encryption (HE), the order of encryption and computation can be interchanged.

Suppose you have data a and b. You can perform computation on the data and then encrypt the result, denoted by E(a*b).

Alternatively, you could encrypt the data and then perform computation, denoted by E(a) * E(b) in the figure below. If the encryption is homomorphic, then these two values, E(a*b) and E(a) * E(b) decrypt to the same value.

Therefore, we can choose to encrypt private data a and b,
outsource computation, say, to the cloud, and decrypt the obtained result of the computation to view the actual meaningful results.

Understanding Homomorphic Encryption intuitively through Homer-morphic Encryption

Let's try to think of a fictional character and draw a relatable analogy.

Remember Homer Simpson from 'The Simpsons'?🙂

The following illustration is aimed at giving an intuitive explanation to what Homomorphic Encryption does.

Let us say you need to get a jewel made and you have your gold ready! You'd now like to call your jeweler (Homer Simpson) and get your jewel made, however you're not very sure if your jeweler is trustworthy. Here's a suggestion.

You may place your gold in a box, lock it and keep the key with yourself. You may now invite over your jeweler and ask him to work on the gold nuggets through the glove box. Once the jewel is done, you may unlock your box and retrieve your jewel. Isn't that cool?

Let us try to parse the analogy a bit.

Your private data is analogous to gold,
outsourcing computations on encrypted data in a public environment is similar to getting your jeweler to work on the gold through glove box, and
decrypting the results of computation to view the meaningful results is analogous to opening the box to get your jewel after the jeweler has left. 🙂

Without delving deep into the math involved, the high-level idea behind homomorphic encryption is as follows.

Homomorphic Encryption uses lattice-based encoding.
Encryption adds noise to a secret inner product.
Decryption subtracts the secret inner product and the noise becomes easy to cancel.

In the next section, let's see the capabilities of Microsoft SEAL, a library for Homomorphic Encryption.

Microsoft SEAL (Simple Encrypted Arithmetic Library)

SEAL (Simple Encrypted Arithmetic Library) is Microsoft's library for Homomorphic Encryption (HE), widely adopted by research teams worldwide. It was first released publicly in 2015, followed by the standardization of HE in November 2018. Microsoft SEAL is available for download and use here.

In recent years, availability of hardware accelerators has also enabled several orders of magnitude speedup. Here's a timeline of how Homomorphic Encryption has been adopted due to advances in research and easier access to compute.

Idea: Computation on encrypted data without decrypting it
2009: Considered impractical due to substantial overhead involved
2011: Breakthrough at Microsoft Research
Subsequent years of research: Practical encoding techniques that achieved several orders of speed-up
2016: Crypto Nets at ICML 2016 - Neural Net predictions on encrypted data

Now that we're familiar with how HE has been adopted over the years, the subsequent sections will focus on the possible applications of HE.

Cloud Compute Scenarios benefitting from HE

The following are some of the cloud computing scenarios that could potentially benefit from Homomorphic Encryption (HE):

Private Storage and Computation
Private Prediction Services
Hosted Private Training
Private Set Intersection
Secure Collaborative Computation

The markets benefitting from such services include healthcare, pharmaceutical, financial, government, insurance, manufacturing, oil and gas sector, to name a few. A few applications of Private AI across industries are listed below.

Finance: Fraud detection, automated claims processing, threat intelligence, and data analytics are some applications.
Healthcare: The scope of Private AI in healthcare includes medical diagnosis, medical support systems such as healthcare bots, preventive care and data analytics.
Manufacturing: Predictive maintenance and data analytics on potentially sensitive data.

Azure ML Private Prediction

An image classification model for encrypted inferencing in Azure Container Instance (ACI), built on Microsoft SEAL, was announced at Microsoft Build Conference 2020. The tutorial can be accessed here.

References

[1] Recording of the PriCon talk

DEV Community: Bala Priya C

A Step-by-Step Starter Kit for Building a Data Quality Framework

Step 1: Define What "Quality" Means for Your Organization

Step 2: Identify and Prioritize Your Critical Data Elements

Step 3: Baseline Current Quality

Step 4: Establish Ownership

Step 5: Define and Implement Quality Rules

Step 6: Build the Response Process

Step 7: Communicate, Report, and Iterate

Conclusion

7 Signs Your Data Quality Framework Is Broken

1. Your Quality Metrics Are Green but Business Complaints Are Constant

2. No One Can Name the Owner of a Quality Problem

3. Rules Are Defined But No One Reviews Exceptions

4. Quality Standards Were Set Once and Never Revisited

5. Downstream Systems Have Their Own Duplicate Quality Checks

6. Data Quality Is Treated as a Data Team Problem

7. There Is No Feedback Loop From Production Issues to Prevention

Final Thoughts

Descriptive Statistics with Python for Beginner Data Scientists

Prerequisites

Table of Contents

What Is Descriptive Statistics?

The Dataset

Measures of Central Tendency

Mean

Median

Visualizing Mean vs Median

Measures of Spread

Standard Deviation

Variance

Range and IQR

Visualizing Spread: The Box Plot

The Five-Number Summary

Skewness

Visualizing Skewness: The Histogram

Putting It All Together

Which Chart for Which Job?

Summary

What's Next?

How to Work with Parquet Files in Python – A Guide with Examples

Table of Contents

What Is the Parquet File Format?

Prerequisites

Writing Parquet Files in Python

Reading Parquet Files in Python

Reading Specific Columns

Writing Parquet Files with Compression

Working with Row Groups and Filtering

A Practical Example: Analyzing Sales Data

When Should You Use Parquet?

Conclusion

5 DIY Python Decorators for Building Cleaner Data Pipelines

Prerequisites

How Python Decorators Work

1. @retry — Handle Transient Failures

2. @timer — Measure Execution Time

3. @validate_schema — Catch Bad Data Early

4. @cache_result — Skip Redundant Computation

5. @log_step — Build a Trail

Wrapping Up

MIT's Missing Semester Class: Beyond the CS Curriculum

So What Does This Course Teach?

#1 – Shell and Shell Scripting

#2 – Working with Vim

#3 – Data Wrangling

#4 – Command-Line Environment

#5 – Version Control with Git

#6 – Debugging and Profiling

#7 – Build Systems, Dependency Management, and More

#8 – Security and Cryptography

#9 – Potpourri

Wrapping Up

How To Code a Simple Number Guessing Game in Python

How Does the Number Guessing Game Work?

Coding the Number Guessing Game

Step 1 - Import the random module

Step 2 - Set up the range and the maximum number of attempts

Step 3 - Generate a random number

Step 4 - Read in the user's input

1. `@retry` — Handle Transient Failures

2. `@timer` — Measure Execution Time

3. `@validate_schema` — Catch Bad Data Early

4. `@cache_result` — Skip Redundant Computation

5. `@log_step` — Build a Trail

Step 1 - Import the `random` module

Objective (`optional`)