DEV Community: Bala Priya C

Descriptive Statistics with Python for Beginner Data Scientists

Bala Priya C — Wed, 11 Mar 2026 07:37:03 +0000

You've probably heard that data scientists spend most of their time cleaning and exploring data... not building fancy models. That's true, and descriptive statistics is the foundation of that exploration.

Before you can ask "what does this data predict?", you need to ask "what does this data look like?" Descriptive statistics gives you the tools to answer that. It tells you where your data is centered, how spread out it is, and whether it's shaped in a way that might cause problems for the models you'll build later.

In this article, you'll work through the core concepts of descriptive statistics using Python, pandas, and matplotlib. Along the way you'll build intuition — not just know which function to call, but understand what the numbers are actually telling you.

If you'd like, you can follow along with the Google Colab notebook.

Prerequisites

Before reading this article, you should be comfortable with:

Basic Python (variables, lists, loops, functions)
What a DataFrame is in pandas (you don't need to be an expert; knowing pd.DataFrame() and df.head() is enough)
Installing packages with pip

You do not need any prior statistics knowledge. We'll build everything from scratch.

What Is Descriptive Statistics?
The Dataset
Measures of Central Tendency — Mean and Median
Measures of Spread — Std Dev, Variance, IQR
The Five-Number Summary
Skewness — Is Your Distribution Symmetric?
Putting It All Together
Which Chart for Which Job?
Summary

What Is Descriptive Statistics?

Descriptive statistics is the practice of summarizing and describing the features of a dataset. Unlike inferential statistics — which draws conclusions about a larger population from a sample — descriptive statistics just tells you about the data you actually have in front of you.

Think of it this way: if a colleague hands you a spreadsheet of 10,000 sales transactions and asks "what do you see?", you can't read every row. Descriptive statistics is how you compress that spreadsheet into a handful of numbers that tell the real story.

There are three categories of descriptive statistics you'll use often:

Category	What it answers	Examples
Central tendency	Where is the "middle" of my data?	mean, median, mode
Spread / variability	How spread out is the data?	variance, std dev, range, IQR
Shape	What does the distribution look like?	skewness, kurtosis

Let's work through each one.

The Dataset

We'll use sales records from 12 retail store branches across a single quarter. Each row represents one branch's performance: total revenue, units sold, and product return rate.

This is small enough to reason about by hand but realistic enough to make the statistics meaningful.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

data = {
    "branch": ["Atlanta", "Boston", "Chicago", "Dallas",
               "Denver", "Houston", "Miami", "Nashville",
               "Phoenix", "Portland", "Seattle", "Tampa"],
    "revenue_usd": [142000, 198000, 175000, 161000,
                    134000, 189000, 210000, 123000,
                    155000, 168000, 202000, 139000],
    "units_sold": [940, 1280, 1105, 1020,
                   870, 1190, 1340, 790,
                   985, 1060, 1295, 855],
    "return_rate_pct": [3.1, 2.4, 2.9, 4.2,
                        3.8, 2.1, 1.9, 5.1,
                        3.4, 2.7, 2.2, 4.6]
}

df = pd.DataFrame(data)
print(df.head())

Output:

    branch  revenue_usd  units_sold  return_rate_pct
0  Atlanta       142000         940              3.1
1   Boston       198000        1280              2.4
2  Chicago       175000        1105              2.9
3   Dallas       161000        1020              4.2
4   Denver       134000         870              3.8

Measures of Central Tendency

Central tendency tells you the "typical" value in your data — the value around which everything else clusters.

Mean

The mean is the sum of all values divided by the count. It's the most commonly used measure of center, but it has a weakness: it is sensitive to outliers.

If one branch had a super good quarter at $900,000, the mean would shoot up and no longer represent the typical branch.

Median

The median is the middle value when your data is sorted. Half the values fall below it, half above.

Because it only cares about position — and not magnitude — extreme values don't move it. This makes it more robust than the mean.

A useful rule of thumb: if mean ≠ median, something interesting is going on. It could be outliers, skewness, or both.

Let's compute both:

revenue = df["revenue_usd"]

mean_rev   = revenue.mean()
median_rev = revenue.median()

print(f"Mean revenue:   ${mean_rev:,.0f}")
print(f"Median revenue: ${median_rev:,.0f}")

Output:

Mean revenue:   $166,333
Median revenue: $164,500

The two values are close — only $1,833 apart. That tells us no single branch is substantially distorting the average. The data is reasonably balanced.

Visualizing Mean vs Median

A bar chart with reference lines is the clearest way to see this relationship in action. Bars below the mean are colored blue, above it green, so you can immediately see which branches are performing above and below average.

fig, ax = plt.subplots(figsize=(10, 5))

colors = ["#4a90d9" if v < mean_rev else "#2ecc71"
          for v in df["revenue_usd"]]

ax.bar(df["branch"], df["revenue_usd"] / 1000,
       color=colors, edgecolor="white", linewidth=0.8)

ax.axhline(mean_rev / 1000, color="#e74c3c", linewidth=2,
           linestyle="--", label=f"Mean: ${mean_rev/1000:.0f}k")
ax.axhline(median_rev / 1000, color="#f39c12", linewidth=2,
           linestyle="-.", label=f"Median: ${median_rev/1000:.0f}k")

ax.set_title("Q3 Revenue by Branch with Mean and Median",
             fontsize=14, fontweight="bold")
ax.set_xlabel("Branch")
ax.set_ylabel("Revenue (USD thousands)")
ax.legend()
ax.tick_params(axis="x", rotation=45)
plt.tight_layout()
plt.savefig("revenue_central_tendency.png", dpi=150)
plt.show()

Output:

In the above dataset, Nashville is the branch with the lowest sales. Miami, Boston, and Seattle lead. The two reference lines sitting almost on top of each other confirms: no outlier is pulling the average off-center.

Measures of Spread

Two datasets can have identical means and still look completely different. One might have all values clustered tightly around the center; another might be all over the place. Measures of spread capture that difference.

Standard Deviation

Standard deviation tells you the typical distance of a data point from the mean. A small std dev means values are packed tightly together. A large one means they're spread out widely.

It's expressed in the same units as your data — dollars here — which makes it easy to interpret directly.

Variance

Variance is the square of the standard deviation. It's used heavily in statistical formulas behind the scenes, but its units are squared (dollars²), which makes it awkward to communicate. Day-to-day, stick with std dev.

Range and IQR

The range (max − min) is simple but not quite robust because one extreme value changes it completely.

The IQR (interquartile range) is the distance between the 25th and 75th percentiles. It describes the spread of the middle 50% of your data. Because it ignores the top and bottom 25%, it's resistant to outliers — just like the median.

std_rev   = revenue.std()
var_rev   = revenue.var()
range_rev = revenue.max() - revenue.min()
iqr_rev   = revenue.quantile(0.75) - revenue.quantile(0.25)

print(f"Std deviation: ${std_rev:,.0f}")
print(f"Variance:      ${var_rev:,.0f}")
print(f"Range:         ${range_rev:,.0f}")
print(f"IQR:           ${iqr_rev:,.0f}")

Output:

Std deviation: $28,908
Variance:      $835,696,970
Range:         $87,000
IQR:           $50,000

The typical branch deviates from the mean by about $29k. The middle 50% of branches span a $50k revenue range. These are the numbers you'd quote in a report. The variance of $831 million — while mathematically correct — would confuse anyone you hand it to.

Visualizing Spread: The Box Plot

The box plot helps visualize the spread of the data. It encodes five numbers in one chart: the minimum, Q1, median, Q3, and maximum.

The box represents the IQR.
The line inside the box is the median. Points beyond the whiskers are potential outliers.

fig, axes = plt.subplots(1, 3, figsize=(12, 5))

cols   = ["revenue_usd", "units_sold", "return_rate_pct"]
labels = ["Revenue (USD)", "Units Sold", "Return Rate (%)"]
colors = ["#4a90d9", "#2ecc71", "#e67e22"]

for ax, col, label, color in zip(axes, cols, labels, colors):
    bp = ax.boxplot(df[col], patch_artist=True, widths=0.5,
                    medianprops=dict(color="black", linewidth=2))
    bp["boxes"][0].set_facecolor(color)
    bp["boxes"][0].set_alpha(0.7)
    ax.set_title(label, fontsize=12, fontweight="bold")
    ax.set_xticks([])

plt.suptitle("Spread of Key Metrics Across Branches",
             fontsize=14, fontweight="bold", y=1.02)
plt.tight_layout()
plt.savefig("boxplots_spread.png", dpi=150)
plt.show()

Output:

The Five-Number Summary

The five-number summary — min, Q1, median, Q3, max — gives you a compact, complete picture of a column's spread. In pandas, describe() gives you this plus the mean and standard deviation for every numeric column at once.

Make this the first thing you run on any new dataset:

print(df[["revenue_usd", "units_sold", "return_rate_pct"]].describe().round(2))

Output:

       revenue_usd  units_sold  return_rate_pct
count        12.00       12.00            12.00
mean     166333.33     1060.83             3.20
std       28908.42      184.29             1.04
min      123000.00      790.00             1.90
25%      141250.00      922.50             2.35
50%      164500.00     1040.00             3.00
75%      191250.00     1212.50             3.90
max      210000.00     1340.00             5.10

A few things stand out:

The return rate range (1.9% to 5.1%) is wide relative to its mean of 3.2%
Revenue mean ($166k) and median ($164.5k) are close — no major outlier distortion
The count row is your quick data quality check: if any column shows fewer rows than expected, you have missing data

Skewness

Skewness measures whether your distribution is symmetric around the mean or has a longer "tail" pulling to one side.

Skewness ≈ 0 — roughly symmetric; the normal distribution is the classic example
Skewness > 0 — right-skewed; a few very high values stretch the tail to the right
Skewness < 0 — left-skewed; a few very low values stretch the tail to the left

Why does this matter? Many statistical tests — and some machine learning algorithms — assume your data is roughly symmetric. If it isn't, results can be unreliable. Knowing skewness early lets you decide whether to transform the data before going further.

for col in ["revenue_usd", "units_sold", "return_rate_pct"]:
    skew = df[col].skew()
    print(f"{col:<25} skewness = {skew:.3f}")

Output:

revenue_usd               skewness = 0.095
units_sold                skewness = 0.167
return_rate_pct           skewness = 0.549

Visualizing Skewness: The Histogram

Histograms are the best way to see skewness. The shape of the bars tells you where data clusters and where it trails off. Overlaying the mean and median makes the direction of any skew immediately visible — when they're apart, the tail is on the side of the mean.

fig, axes = plt.subplots(1, 3, figsize=(13, 4))

cols   = ["revenue_usd", "units_sold", "return_rate_pct"]
labels = ["Revenue (USD)", "Units Sold", "Return Rate (%)"]
colors = ["#4a90d9", "#2ecc71", "#e67e22"]

for ax, col, label, color in zip(axes, cols, labels, colors):
    ax.hist(df[col], bins=6, color=color, edgecolor="white",
            linewidth=0.8, alpha=0.85)

    mean_val   = df[col].mean()
    median_val = df[col].median()
    ax.axvline(mean_val,   color="#e74c3c", linestyle="--",
               linewidth=1.8, label="Mean")
    ax.axvline(median_val, color="black",   linestyle="-.",
               linewidth=1.8, label="Median")

    skew_val = df[col].skew()
    ax.set_title(f"{label}\nskew = {skew_val:.3f}",
                 fontsize=11, fontweight="bold")
    ax.legend(fontsize=9)

plt.suptitle("Histograms with Mean vs Median — Detecting Skewness",
             fontsize=13, fontweight="bold", y=1.03)
plt.tight_layout()
plt.savefig("histograms_skewness.png", dpi=150)
plt.show()

Output:

Look at the return rate panel on the right. The mean (red dashed line) sits to the right of the median (black dash-dot). That gap is the visual fingerprint of positive skew. Revenue and units sold, by contrast, are not as skewed.

Putting It All Together

Here's a reusable function that prints the key descriptive stats for any numeric column:

def summarize(series, label):
    print(f"\n── {label} ──")
    print(f"  Mean:    {series.mean():>10.2f}")
    print(f"  Median:  {series.median():>10.2f}")
    print(f"  Std Dev: {series.std():>10.2f}")
    print(f"  IQR:     {series.quantile(0.75) - series.quantile(0.25):>10.2f}")
    print(f"  Skew:    {series.skew():>10.3f}")

summarize(df["revenue_usd"],     "Revenue (USD)")
summarize(df["units_sold"],      "Units Sold")
summarize(df["return_rate_pct"], "Return Rate (%)")

Output:

── Revenue (USD) ──
  Mean:     166333.33
  Median:   164500.00
  Std Dev:   28908.42
  IQR:       50000.00
  Skew:         0.095

── Units Sold ──
  Mean:       1060.83
  Median:     1040.00
  Std Dev:     184.29
  IQR:         290.00
  Skew:         0.167

── Return Rate (%) ──
  Mean:          3.20
  Median:        3.00
  Std Dev:       1.04
  IQR:           1.55
  Skew:         0.549

Drop this function into any exploratory notebook and run it on every column before you analyze the data further.

Which Chart for Which Job?

Here's an overview of which chart you should use:

What you want to show	Best chart	Key method
Distribution shape and skewness	Histogram	`ax.hist()`
Spread, median, and outliers	Box plot	`ax.boxplot()`
Comparing values across categories	Bar chart	`ax.bar()`
Mean vs median on a bar chart	Bar + reference lines	`ax.axhline()`

Summary

Concept	What it tells you	pandas method
Mean	Average value	`.mean()`
Median	Middle value — outlier-robust	`.median()`
Std deviation	Typical distance from the mean	`.std()`
IQR	Spread of the middle 50%	`.quantile()`
Skewness	Symmetry of the distribution	`.skew()`
Five-number summary	Full spread at a glance	`.describe()`

What's Next?

In this article you learned to describe your data with numbers and charts. But there's a deeper question behind all of this: why do values cluster and spread the way they do? The answer lies in probability distributions, and that's exactly what we'll cover in the next article in this series.

How to Work with Parquet Files in Python – A Guide with Examples

Bala Priya C — Thu, 05 Mar 2026 09:06:19 +0000

If you've spent time in data engineering or analytics, you've almost certainly run into Parquet files. They show up everywhere — in data lakes, machine learning pipelines, cloud storage buckets, and more.

But what makes Parquet special, and how do you actually work with it in Python?

In this tutorial, I'll walk you through reading, writing, filtering, and compressing Parquet files using Python. By the end, you'll have a solid foundation for using Parquet in your own data pipelines.

You can get the code on GitHub.

What Is the Parquet File Format?
Prerequisites
Writing Parquet Files in Python
Reading Parquet Files in Python
Reading Specific Columns
Writing Parquet Files with Compression
Working with Row Groups and Filtering
A Practical Example: Analyzing Sales Data
When Should You Use Parquet?

What Is the Parquet File Format?

Parquet is an open-source columnar storage format originally developed by Cloudera and Twitter, and now part of the Apache ecosystem.

Unlike CSV or JSON, which store data row by row, Parquet stores data column by column. That might seem like a small implementation detail, but it has a big impact on performance.

Here's why Parquet is widely used:

Efficient compression — because all values in a column share the same data type, they compress far better than mixed-type rows
Column pruning — you can read only the columns you need, skipping the rest entirely
Schema preservation — data types are stored in the file, so integers don't become strings on read (and other subtle bugs)
Broad ecosystem support — works natively with Spark, Pandas, DuckDB, BigQuery, Athena, and more

Parquet is often compared to ORC (or optimized row columnar). Both are columnar formats, but Parquet has wider cross-platform support and tends to be the default choice outside of Hadoop/Hive environments.

Prerequisites

Before we get started, make sure you have:

Python 3.10 or a later version installed
Basic familiarity with pandas DataFrames
Some understanding of file I/O in Python

You'll need to install two libraries:

pip install pyarrow pandas

We'll use PyArrow as the engine for reading and writing Parquet files. You can also use [fastparquet](https://fastparquet.readthedocs.io/en/latest/) as an alternative engine, but PyArrow is the more commonly recommended choice.

Writing Parquet Files in Python

Let's start by creating a Parquet file. We'll build a small product inventory dataset and write it to disk.

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

# Create a product inventory dataset
data = {
    'product_id': [1001, 1002, 1003, 1004, 1005],
    'product_name': ['Wireless Mouse', 'Mechanical Keyboard', 'USB-C Hub', 'Monitor Stand', 'Webcam HD'],
    'category': ['Peripherals', 'Peripherals', 'Accessories', 'Accessories', 'Peripherals'],
    'price': [29.99, 89.99, 49.99, 34.99, 74.99],
    'stock': [150, 80, 200, 95, 60]
}

df = pd.DataFrame(data)

# Convert to PyArrow Table and write as Parquet
table = pa.Table.from_pandas(df)
pq.write_table(table, 'inventory.parquet')

print("Parquet file created successfully!")

Output:

Parquet file created successfully!

Here's what's happening step by step. We start with a standard pandas DataFrame. Then we convert it to a PyArrow Table using pa.Table.from_pandas(). This is PyArrow's in-memory columnar representation. Finally, pq.write_table() serializes it to disk in Parquet format.

The conversion step is necessary because Parquet is a columnar format and PyArrow handles the translation between pandas' row-oriented structure and Parquet's column-oriented storage.

Reading Parquet Files in Python

Now let's read the file back:

# Read the Parquet file
table = pq.read_table('inventory.parquet')

# Convert to pandas for easy viewing
df_read = table.to_pandas()

print(df_read)
print(f"\nData types:\n{df_read.dtypes}")

Output:

   product_id         product_name     category  price  stock
0        1001       Wireless Mouse  Peripherals  29.99    150
1        1002  Mechanical Keyboard  Peripherals  89.99     80
2        1003            USB-C Hub  Accessories  49.99    200
3        1004        Monitor Stand  Accessories  34.99     95
4        1005            Webcam HD  Peripherals  74.99     60

Data types:
product_id        int64
product_name     object
category         object
price           float64
stock             int64
dtype: object

pq.read_table() loads the Parquet file into a PyArrow Table. We then call .to_pandas() to convert it into a familiar DataFrame.

Notice that data types are preserved correctly — price is float64, not a string. This is one of Parquet's practical advantages over CSV, where you often have to explicitly cast types after reading.

Reading Specific Columns

One of Parquet's most powerful features is the ability to read only the columns you actually need. This is called column pruning, and it can substantially reduce memory usage and load time on large files.

# Read only product name and price
table_subset = pq.read_table('inventory.parquet', columns=['product_name', 'price'])
df_subset = table_subset.to_pandas()

print(df_subset)

Output:

          product_name  price
0       Wireless Mouse  29.99
1  Mechanical Keyboard  89.99
2            USB-C Hub  49.99
3        Monitor Stand  34.99
4            Webcam HD  74.99

By passing a columns list to read_table(), PyArrow only reads those two columns from disk — the rest are skipped entirely. If your file has 100 columns but your query only needs 4, you're doing roughly 4% of the I/O work. That adds up fast at scale.

Writing Parquet Files with Compression

Parquet supports several compression codecs. Let's compare them on a larger dataset:

import os

# Generate a larger dataset
large_data = {
    'transaction_id': range(50000),
    'store': ['Downtown', 'Uptown', 'Suburban', 'Airport', 'Mall'] * 10000,
    'product': [f'product_{i % 200}' for i in range(50000)],
    'amount': [round(10 + (i % 500) * 0.5, 2) for i in range(50000)]
}

df_large = pd.DataFrame(large_data)
table_large = pa.Table.from_pandas(df_large)

# Write with different compression codecs
for codec in ['NONE', 'SNAPPY', 'GZIP', 'ZSTD']:
    path = f'transactions_{codec.lower()}.parquet'
    pq.write_table(table_large, path, compression=codec)
    size = os.path.getsize(path)
    print(f"{codec:<8}: {size:>10,} bytes")

Output:

NONE    :    635,565 bytes
SNAPPY  :    316,033 bytes
GZIP    :    177,654 bytes
ZSTD    :    158,082 bytes

Each codec makes a different trade-off between file size and speed:

NONE — no compression, fastest write, largest file
SNAPPY — fast compression and decompression, moderate size reduction, good for hot data
GZIP — better compression than Snappy, slower, good for archival
ZSTD — best compression ratio with competitive speed, generally the recommended default

For most production use cases, ZSTD is a solid default. It gives you the smallest files without a significant speed penalty.

Working with Row Groups and Filtering

Parquet files are internally divided into row groups — chunks of rows stored together. This structure enables a powerful optimization called predicate pushdown, where Parquet skips entire row groups that can't possibly match your filter condition.

Let's write a file with explicit row group sizing and then filter it:

# Write with smaller row groups (useful for filtering large files)
pq.write_table(
    table_large,
    'transactions_grouped.parquet',
    compression='ZSTD',
    row_group_size=10000
)

# Use PyArrow's dataset API for efficient filtering
import pyarrow.dataset as ds

dataset = ds.dataset('transactions_grouped.parquet', format='parquet')

# Filter: only transactions from the Downtown store over $150
filtered = dataset.to_table(
    filter=(ds.field('store') == 'Downtown') & (ds.field('amount') > 150)
)

df_filtered = filtered.to_pandas()
print(f"Matching transactions: {len(df_filtered)}")
print(df_filtered.head())

Output:

Matching transactions: 4300
   transaction_id     store      product  amount
0             285  Downtown   product_85   152.5
1             290  Downtown   product_90   155.0
2             295  Downtown   product_95   157.5
3             300  Downtown  product_100   160.0
4             305  Downtown  product_105   162.5

The ds.dataset() API with a filter argument doesn't load the entire file into memory. Instead, it evaluates the filter against each row group's metadata and skips groups that can't contain matching rows. The actual rows are only read when you call .to_table(). This is particularly useful for large files.

A Practical Example: Analyzing Sales Data

Let's put it all together with a better example. Say you receive daily sales exports and need to store them efficiently and query them by region and date.

from datetime import datetime, timedelta
import random

# Simulate 30 days of sales data
regions = ['North', 'South', 'East', 'West']
products = ['Laptop', 'Tablet', 'Headphones', 'Charger', 'Case']

records = []
start = datetime(2025, 1, 1)

for day in range(30):
    date = start + timedelta(days=day)
    for _ in range(200):
        records.append({
            'date': date.strftime('%Y-%m-%d'),
            'region': random.choice(regions),
            'product': random.choice(products),
            'units_sold': random.randint(1, 20),
            'revenue': round(random.uniform(50, 2000), 2)
        })

df_sales = pd.DataFrame(records)
table_sales = pa.Table.from_pandas(df_sales)

# Save as compressed Parquet
pq.write_table(table_sales, 'sales_jan2025.parquet', compression='ZSTD')
print(f"Saved {len(df_sales):,} records to Parquet")

# Query: total revenue by region for the first two weeks
dataset = ds.dataset('sales_jan2025.parquet', format='parquet')

first_two_weeks = dataset.to_table(
    columns=['region', 'revenue', 'date'],
    filter=ds.field('date') <= '2025-01-14'
).to_pandas()

summary = first_two_weeks.groupby('region')['revenue'].sum().sort_values(ascending=False)
print(f"\nRevenue by region (Jan 1–14):")
print(summary.round(2))

Output:

Saved 6,000 records to Parquet

Revenue by region (Jan 1–14):
region
West     752919.48
North    739783.22
South    730002.82
East     692128.58
Name: revenue, dtype: float64

This is a pattern you'll see in real data engineering work. Parquet handles the storage efficiently, column pruning limits what gets loaded into memory, and predicate pushdown ensures only the relevant date range is scanned.

When Should You Use Parquet?

Use Parquet when you:

Work with analytical or reporting workloads where you query specific columns
Need efficient, compressed storage for medium-to-large datasets
Store data in cloud platforms (S3, GCS, Azure Blob) that are billed by bytes scanned
Use tools like Spark, DuckDB, Athena, or BigQuery that have native Parquet support
Want schema enforcement across your pipeline

Don't use Parquet when you:

Need human-readable files; use CSV or JSON for that
Are doing frequent row-level updates or inserts; Parquet is write-once, read-many
Work with very small datasets where the overhead isn't worth it
Need streaming row-by-row writes; consider Avro instead

Conclusion

I hope you found this tutorial useful. Parquet's columnar layout, built-in compression, and schema preservation add up to faster queries, smaller files, and fewer type-related bugs.

In this tutorial, you learned how to write and read Parquet files, use column pruning to reduce I/O, apply compression codecs, and filter large files efficiently using predicate pushdown. These are the building blocks you'll reach for repeatedly in any data-heavy Python project.

Try swapping your next CSV export for Parquet and see how it affects your pipeline. Happy coding!

5 DIY Python Decorators for Building Cleaner Data Pipelines

Bala Priya C — Mon, 23 Feb 2026 07:30:20 +0000

Data pipelines often tend to accumulate the same boilerplate in a lot of places. You'll likely have retry logic copy-pasted across functions, timing code scattered through the codebase, validation checks buried inside business logic, and more. Decorators are a clean way to pull that stuff out.

They let you separate what a function does from how it behaves, and in data engineering, that's a useful separation to make.

This article walks through five practical decorators you can use to keep your data pipelines maintainable.

You can get the code on GitHub.

Prerequisites

Before we get started, you should be comfortable with:

Python functions as first-class objects — functions can be passed as arguments and returned from other functions
*args and **kwargs — used throughout to make decorators flexible enough to wrap any function signature
Basic exception handling — try/except/raise blocks appear in several decorators
functools module — specifically functools.wraps, explained in the next section

You don't need to have written a decorator for data pipelines before. That's what this article is for.

How Python Decorators Work

A decorator is a function that takes another function as input and returns a new function as output. The @ symbol is syntactic sugar to make it read cleanly.

Here's the simplest possible example:

def shout(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        return str(result).upper()
    return wrapper

@shout
def greet(name):
    return f"hello, {name}"

print(greet("world"))
# HELLO, WORLD

When Python sees @shout above greet, it does this behind the scenes:

greet = shout(greet)

So calling greet("world") calls wrapper("world"), which calls the original greet, gets "hello, world", and returns "HELLO, WORLD". The original function is unchanged; it's just been wrapped.

The wrapper uses *args and **kwargs so it can forward any arguments without knowing them in advance. This is what makes decorators reusable across functions with different signatures.

Note: without @functools.wraps(func), your decorated function loses its original name and docstring. greet.__name__ would show wrapper instead of greet, which makes debugging and log output harder. Every decorator in this article includes functools.wraps for this reason.

1. `@retry` — Handle Transient Failures

In many projects, database connections and API calls often fail, a file is briefly locked by another process, and more. The common fix is wrapping every external call in a for loop with a try/except. But copy-pasting that pattern across your codebase gets messy fast.

Here's a reusable @retry decorator:

import time
import functools

def retry(max_attempts=3, delay=2, exceptions=(Exception,)):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    if attempt == max_attempts:
                        raise
                    print(f"Attempt {attempt} failed: {e}. Retrying in {delay}s...")
                    time.sleep(delay)
        return wrapper
    return decorator

Here we simulate a flaky SQLite read that fails the first two times before succeeding on the third attempt:

import sqlite3

# Setup: create an in-memory DB with some orders
conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE orders (order_id, customer_id, amount, status)")
conn.executemany("INSERT INTO orders VALUES (?, ?, ?, ?)", [
    (1, 101, 59.99,  "completed"),
    (2, 102, 120.00, "completed"),
    (3, 101, 34.50,  "refunded"),
    (4, 103, 89.00,  "completed"),
])
conn.commit()

# Simulate a connection that fails twice before succeeding
attempt_count = 0

@retry(max_attempts=3, delay=1, exceptions=(sqlite3.OperationalError,))
def fetch_completed_orders(conn):
    global attempt_count
    attempt_count += 1
    if attempt_count < 3:
        raise sqlite3.OperationalError("database is locked")
    cursor = conn.execute("SELECT * FROM orders WHERE status = 'completed'")
    return cursor.fetchall()


rows = fetch_completed_orders(conn)

print(rows)

Output:

Attempt 1 failed: database is locked. Retrying in 1s...
Attempt 2 failed: database is locked. Retrying in 1s...
[(1, 101, 59.99, 'completed'), (2, 102, 120.0, 'completed'), (4, 103, 89.0, 'completed')]

We're specific about sqlite3.OperationalError that covers locks and connection issues. A sqlite3.ProgrammingError from a malformed query is not worth retrying; it'll fail the same way every time.

There are three nested functions here because retry takes configuration arguments. retry(...) returns decorator, which is the actual decorator. decorator receives the function and returns wrapper.

2. `@timer` — Measure Execution Time

When a pipeline run is slower than expected, the naive approach is sprinkling time.time() calls through your code. It works, but timing logic ends up tangled with business logic everywhere, and you have to clean it all up when you're done. Let's write a @timer decorator:

import time
import functools

def timer(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        result = func(*args, **kwargs)
        elapsed = time.perf_counter() - start
        print(f"[TIMER] {func.__name__} completed in {elapsed:.4f}s")
        return result
    return wrapper

Here we apply it to a transformation step that computes customer lifetime value and assigns spend tiers across a list of orders:

orders = [
    {"order_id": 1, "customer_id": 101, "amount": 59.99,  "status": "completed"},
    {"order_id": 2, "customer_id": 102, "amount": 120.00, "status": "completed"},
    {"order_id": 3, "customer_id": 101, "amount": 34.50,  "status": "refunded"},
    {"order_id": 4, "customer_id": 103, "amount": 89.00,  "status": "completed"},
    {"order_id": 5, "customer_id": 102, "amount": 45.00,  "status": "completed"},
]

@timer
def transform_orders(orders: list[dict]) -> list[dict]:
    # Compute lifetime value per customer
    ltv = {}
    for o in orders:
        if o["status"] == "completed":
            ltv[o["customer_id"]] = ltv.get(o["customer_id"], 0) + o["amount"]

    # Enrich each order with LTV and spend tier
    def spend_tier(value):
        if value >= 200: return "high"
        if value >= 100: return "mid"
        return "low"

    return [
        {**o, "customer_ltv": ltv.get(o["customer_id"], 0),
              "spend_tier": spend_tier(ltv.get(o["customer_id"], 0))}
        for o in orders
    ]


result = transform_orders(orders)

for row in result:
    print(row)

Output:

[TIMER] transform_orders completed in 0.0001s
{'order_id': 1, 'customer_id': 101, 'amount': 59.99, 'status': 'completed', 'customer_ltv': 59.99, 'spend_tier': 'low'}
{'order_id': 2, 'customer_id': 102, 'amount': 120.0, 'status': 'completed', 'customer_ltv': 165.0, 'spend_tier': 'mid'}
{'order_id': 3, 'customer_id': 101, 'amount': 34.5, 'status': 'refunded', 'customer_ltv': 59.99, 'spend_tier': 'low'}
{'order_id': 4, 'customer_id': 103, 'amount': 89.0, 'status': 'completed', 'customer_ltv': 89.0, 'spend_tier': 'low'}
{'order_id': 5, 'customer_id': 102, 'amount': 45.0, 'status': 'completed', 'customer_ltv': 165.0, 'spend_tier': 'mid'}

Inside wrapper, we store the result before printing the time and return it after, so the decorator is invisible to whatever called the function. We use time.perf_counter() instead of time.time() because it has higher resolution and isn't affected by system clock adjustments.

3. `@validate_schema` — Catch Bad Data Early

Data from upstream sources isn't always consistently shaped. A bug in a producer might start omitting a field. A schema change might rename a key. If a required field is missing, you want to know at the point the record enters your pipeline — not several transformations later when something raises a KeyError with no indication of which record caused it.

import functools

def validate_schema(required_keys):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(data, *args, **kwargs):
            if not isinstance(data, dict):
                raise TypeError(f"Expected dict, got {type(data).__name__}")
            missing = required_keys - data.keys()
            if missing:
                raise ValueError(f"Missing required keys: {missing}")
            return func(data, *args, **kwargs)
        return wrapper
    return decorator

We apply it to normalize_order, which standardizes each raw order record before it's passed to the transformation step:

@validate_schema(required_keys={"order_id", "customer_id", "amount", "status"})
def normalize_order(order: dict) -> dict:
    return {
        "order_id": order["order_id"],
        "customer_id": order["customer_id"],
        "amount": round(float(order["amount"]), 2),
        "status": order["status"].lower().strip(),
        "note": order.get("note", ""),   # optional — fine to be absent
    }


# Valid record — passes through
normalize_order({"order_id": 1, "customer_id": 101, "amount": "59.99", "status": "Completed"})
# {"order_id": 1, "customer_id": 101, "amount": 59.99, "status": "completed", "note": ""}

# Missing amount — caught immediately
normalize_order({"order_id": 2, "customer_id": 102, "status": "Completed"})
# ValueError: Missing required keys: {'amount'}

# Wrong type entirely — also caught
normalize_order([1, 101, 59.99, "completed"])
# TypeError: Expected dict, got list

Two checks run before the function body executes. First, we confirm the input is a dict. Second, we use set subtraction to find any missing required keys and raise a ValueError that names them exactly.

Note that note is not in required_keys. It's optional and we use .get() for it. The decorator only enforces what's truly required.

4. `@cache_result` — Skip Redundant Computation

Some pipeline steps are expensive and return the same result for the same inputs. A lookup table loaded from a database, a CSV read from disk, a slow aggregation — if you're calling these multiple times with the same arguments, there's no reason to recompute it.

Python has functools.lru_cache built in, but it only works with hashable arguments. Dicts and lists — common in data pipelines — won't work with it. This decorator handles those cases by hashing the arguments:

import functools
import hashlib
import json

def cache_result(func):
    cache = {}

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        key = hashlib.md5(
            json.dumps((args, kwargs), sort_keys=True, default=str).encode()
        ).hexdigest()

        if key not in cache:
            cache[key] = func(*args, **kwargs)
            print(f"[CACHE] Computed result for {func.__name__}")
        else:
            print(f"[CACHE] Cache hit for {func.__name__}")

        return cache[key]

    wrapper.cache_clear = lambda: cache.clear()
    return wrapper

We apply it to load_discount_codes, which reads a reference table from our SQLite database. This table is queried repeatedly during enrichment but rarely changes within a run:

import sqlite3

conn = sqlite3.connect(":memory:")
conn.execute("CREATE TABLE discount_codes (code TEXT, discount_pct REAL, active INTEGER)")
conn.executemany("INSERT INTO discount_codes VALUES (?, ?, ?)", [
    ("SAVE10", 10.0, 1),
    ("SAVE20", 20.0, 1),
    ("OLD5",    5.0, 0),
])
conn.commit()

@cache_result
def load_discount_codes(active_only: bool = True) -> dict:
    query = "SELECT code, discount_pct FROM discount_codes"
    if active_only:
        query += " WHERE active = 1"
    rows = conn.execute(query).fetchall()
    return {code: pct for code, pct in rows}


# First call hits the database
codes = load_discount_codes(active_only=True)

# Second call is instant
codes = load_discount_codes(active_only=True)

print(codes)

# Different argument = separate cache entry
all_codes = load_discount_codes(active_only=False)

Output:

[CACHE] Computed result for load_discount_codes
[CACHE] Cache hit for load_discount_codes
{'SAVE10': 10.0, 'SAVE20': 20.0}
[CACHE] Computed result for load_discount_codes

The cache key is generated by serializing args and kwargs to JSON, then MD5-hashing the result. sort_keys=True ensures consistent output regardless of keyword argument order; default=str handles types that JSON can't serialize natively. The cache dict lives in the closure and persists for the program's lifetime. cache_clear is attached to the wrapper as an escape hatch if you need to force a fresh load.

5. `@log_step` — Build a Trail

Without structured logging, debugging a failed pipeline run means guesswork. You have no record of what ran, what succeeded, or where things went sideways.

import functools
import logging

logging.basicConfig(level=logging.INFO, format="%(asctime)s — %(message)s")
logger = logging.getLogger(__name__)

def log_step(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        logger.info(f"START  {func.__name__}")
        try:
            result = func(*args, **kwargs)
            logger.info(f"END    {func.__name__} — OK")
            return result
        except Exception as e:
            logger.error(f"FAIL   {func.__name__} — {type(e).__name__}: {e}")
            raise
    return wrapper

We apply it across the main stages of our pipeline:

@log_step
def fetch_completed_orders(conn) -> list[tuple]:
    cursor = conn.execute("SELECT * FROM orders WHERE status = 'completed'")
    return cursor.fetchall()

@log_step
def transform_orders(orders: list[dict]) -> list[dict]:
    ...

@log_step
def write_results(conn, rows: list[dict]) -> int:
    conn.execute("CREATE TABLE IF NOT EXISTS orders_transformed "
                 "(order_id, customer_id, amount, status, customer_ltv, spend_tier)")
    conn.executemany(
        "INSERT INTO orders_transformed VALUES (:order_id, :customer_id, :amount, "
        ":status, :customer_ltv, :spend_tier)",
        rows,
    )
    conn.commit()
    return len(rows)

After logging the error, we call bare raise. This re-raises the original exception so the pipeline still fails. Without it, the decorator would silently swallow every error and execution would continue. Always re-raise in a logging decorator.

Wrapping Up

Decorators let you write that logic once and apply it anywhere. The five patterns here — retry, timer, schema validation, caching, and logging — cover the most common cross-cutting concerns in data pipelines.

A good starting point is @log_step and @retry. They address the most immediate pain points in data pipelines with minimal setup. Add the others as your pipeline grows and the needs become clearer.

Happy coding!

MIT's Missing Semester Class: Beyond the CS Curriculum

Bala Priya C — Fri, 04 Aug 2023 09:51:04 +0000

I was in my second semester of grad school when I learned about MIT’s missing semester class. Over the next few weeks, I worked through the class and found it super helpful. Here's why.

In a conventional CS curriculum, you’ll diligently work through, perhaps, two semester-long classes, perfecting your data structures and algorithms foundations (yeah, also count the weekends of leetcoding). You'll also take classes in compiler design, operating systems, and much more. These are core computer science skills.

But what you don't learn at the university, though, is how to exit Vim (yeah, I know!) or how to git rebase with confidence (add as many as you’d like).

Rightly called The Missing Semester (of Your CS Education), this class from MIT will teach you how to use some of the tools that are fundamental to the software engineering ecosystem. From shell scripting to the fundamentals of information security—spanning around 12 lectures—you can add a bunch of practical skills to your toolbox.

And if you’re into CS, chances are you’ve already heard of or taken this class. If you’ve taken it before, I’m sure you enjoyed the process. But if you haven’t already, happy learning!

So What Does This Course Teach?

Here’s what I did when I took the class (very studently of me):

Watched the lectures on YouTube; took notes and coded along where possible.
Read through the lecture notes on missing.csail.mit.edu.
Completed the exercises for each module.

I’ll briefly go over the topics covered in the lectures/modules.

#1 – Shell and Shell Scripting

From automating ~~boring~~ simple tasks to building ETL pipelines in production, shell scripting is the swiss-army knife of a programmer’s toolbox.

The first lecture is an introduction to shell (and the class). And the second lecture covers shell scripting fundamentals. Be sure to do the exercises.

#2 – Working with Vim

Yes, I had to spend a couple of dedicated days learning my way around Vim. The lecture on Vim covers basics like navigating and editing files. It also covers customizing Vim and advanced features like macros.

#3 – Data Wrangling

To me, this was, perhaps, the most challenging module. Learning about regular expressions and tools like sed and awk—all in a single lecture—was quite overwhelming.

#4 – Command-Line Environment

This lecture introduces the learners to command-line capabilities including process management, all things ssh for working with remote machines, dotfiles, and more.

#5 – Version Control with Git

Even if you are a student, you should learn to use Git. It can be helpful in working on your projects as well as in your internships.

The lecture emphasizes how we can do better than just knowing our way around a few Git commands, and covers Git’s data model. And goes on to cover the usage of basic to advanced commands.

#6 – Debugging and Profiling

Debugging and profiling is one of my favorite modules from the class. From native (or should rather say naïve) print statements to using the shell program logger and Python debugger, this covers reasonable depth. You’ll also learn tracing system calls with strace.

In the profiling part, you will learn profiling for CPU usage, memory, and visualizing profiles to make sense of them better. In addition, you’ll get to know using perf for event profiling and tools for resource monitoring.

#7 – Build Systems, Dependency Management, and More

Originally titled as metaprogramming, this lecture provides a comprehensive overview of build systems, dependency management, continuous integration systems, and testing.

#8 – Security and Cryptography

This lecture is an introduction to hash functions and cryptography basics. One of the easiest lectures where you don't feel overwhelmed at all.

#9 – Potpourri

True to its name, this lecture is a collection of interesting hacks. I particularly liked the overview of daemon processes and keyboard remapping.

Wrapping Up

So if you are a student or a new grad, the tools you learn in this class should be helpful in your software engineering career. Even if you are looking to pivot to software engineering from another domain, you can work through this class after you’ve gained some programming experience.

So have you watched the first lecture yet?

How To Code a Simple Number Guessing Game in Python

Bala Priya C — Thu, 27 Jul 2023 02:30:28 +0000

I spent the last weekend compiling a list of games you can code in Python. But why?

If you're a beginner Python programmer, building fun games will help you learn the language faster—and better—without getting bogged down in the syntax and the like. I built some of these games when I was learning Python; I quite enjoyed the process!

The first game you can code—and the simplest of them all—is a number guessing game (or Guess the Number!). So I thought I'd write a step-by-step tutorial to code this game—and help beginners learn some of the fundamentals along the way.

Let's begin!

How Does the Number Guessing Game Work?

In a number guessing game, the user guesses a randomly generated secret number within a given number of attempts.

After each guess, the user gets hints on whether their guess is too high, too low, or correct. So yeah, the game ends when the user guesses the secret number or runs out of attempts.

Coding the Number Guessing Game

Let's get to coding! Create a new Python script and code along.

Step 1 - Import the `random` module

Let's start by importing the built-in random module. The random module has functions we can use to generate a random secret number within the specified range:

import random

Note: The random module gives pseudo-random numbers—and not truly random numbers. So don't use it for sensitive applications such as password generation.

Step 2 - Set up the range and the maximum number of attempts

Next, we need to decide on the range for the secret number and the maximum number of attempts allowed for the player. For this tutorial, let's set the lower_bound and upper_bound to 1 and 1000, respectively. Also, set the maximum attempts allowed max_attempts to 10:

lower_bound = 1
upper_bound = 1000
max_attempts = 10

Step 3 - Generate a random number

Now, let's generate a random number within the specified range using the random.randint() function. This is the secret number that the user needs to guess:

secret_number = random.randint(lower_bound, upper_bound)

Step 4 - Read in the user's input

To get input from the user, let's create a function called get_guess(). Remember, the user can enter an invalid input: a number outside the range [lower_bound, upper_bound], a string or a floating point number, and more.

We handle this in the get_guess() function that continuously prompts the user to enter a number—within the specified range—until they provide a valid input.

Here, we use a while loop to prompt the user for a valid input until they enter an integer between lower_bound and upper_bound:

def get_guess():
    while True:
        try:
            guess = int(input(f"Guess a number between {lower_bound} and {upper_bound}: "))
            if lower_bound <= guess <= upper_bound:
                return guess
            else:
                print("Invalid input. Please enter a number within the specified range.")
        except ValueError:
            print("Invalid input. Please enter a valid number.")

Step 5 - Validate the user's guess

Next, let's define a check_guess() function that takes the user's guess and the secret number as inputs and provides feedback on whether the guess is correct, too high, or too low.

The function compares the player's guess with the secret number and returns a corresponding message:

def check_guess(guess, secret_number):
    if guess == secret_number:
        return "Correct"
    elif guess < secret_number:
        return "Too low"
    else:
        return "Too high"

Step 6 - Track the number of attempts and detect end-of-game conditions

We'll now create the function play_game() that handles the game logic and puts everything together. The function uses the attempts variable to keep track of the number of attempts made by the user. Within a while loop, the user is prompted to enter a guess that's processed by the get_guess() function.

The call to the check_guess() function provides feedback on the user's guess:

If the guess is correct, the user wins, and the game ends.
Otherwise, the user is given another chance to guess.
And this continues until the player guesses the secret number or runs out of attempts.

Here's the play_game() function:

def play_game():
    attempts = 0
    won = False

    while attempts < max_attempts:
        attempts += 1
        guess = get_guess()
        result = check_guess(guess, secret_number)

        if result == "Correct":
            print(f"Congratulations! You guessed the secret number {secret_number} in {attempts} attempts.")
            won = True
            break
        else:
            print(f"{result}. Try again!")

    if not won:
        print(f"Sorry, you ran out of attempts! The secret number is {secret_number}.")

Step 7 - Play the game!

Finally, you can call the play_game() function every time the Python script is run:

if __name__ == "__main__":
    print("Welcome to the Number Guessing Game!")
    play_game()

Putting it all together

Now our Python script looks like so:

# main.py
import random

# define range and max_attempts
lower_bound = 1
upper_bound = 1000
max_attempts = 10

# generate the secret number
secret_number = random.randint(lower_bound, upper_bound)

# Get the user's guess
def get_guess():
    while True:
        try:
            guess = int(input(f"Guess a number between {lower_bound} and {upper_bound}: "))
            if lower_bound <= guess <= upper_bound:
                return guess
            else:
                print("Invalid input. Please enter a number within the specified range.")
        except ValueError:
            print("Invalid input. Please enter a valid number.")

# Validate guess
def check_guess(guess, secret_number):
    if guess == secret_number:
        return "Correct"
    elif guess < secret_number:
        return "Too low"
    else:
        return "Too high"

# track the number of attempts, detect if the game is over
def play_game():
    attempts = 0
    won = False

    while attempts < max_attempts:
        attempts += 1
        guess = get_guess()
        result = check_guess(guess, secret_number)

        if result == "Correct":
            print(f"Congratulations! You guessed the secret number {secret_number} in {attempts} attempts.")
            won = True
            break
        else:
            print(f"{result}. Try again!")

    if not won:
        print(f"Sorry, you ran out of attempts! The secret number is {secret_number}.")

if __name__ == "__main__":
    print("Welcome to the Number Guessing Game!")
    play_game()

Here's the output from a sample run of the script:

Welcome to the Number Guessing Game!
Guess a number between 1 and 1000: 500
Too low. Try again!
Guess a number between 1 and 1000: 750
Too high. Try again!
Guess a number between 1 and 1000: 625
Too low. Try again!
Guess a number between 1 and 1000: 685
Too low. Try again!
Guess a number between 1 and 1000: 710
Too low. Try again!
Guess a number between 1 and 1000: 730
Congratulations! You guessed the secret number 730 in 6 attempts.

Wrapping Up

Congratulations! You've successfully built a number guessing game in Python. I'll see you all soon in another tutorial. But don't wait for me. Check out other games you can build—features you can code—and start coding!

How To Write an Effective Technical Résumé

Bala Priya C — Tue, 04 Jan 2022 15:41:19 +0000

Getting your résumé writing right is an important step in your developer journey.

Have you ever been a part of the job search process—be it an internship or a full-time opportunity—as a student or a new grad?

If yes, you already know: getting past the résumé screening step, and landing that interview call can often be harder than the actual interview.

Over the next few minutes, you'll get to know some actionable tips for résumé writing, that you could use to revamp your résumé. This post is inspired by Jessie Newman's webinar for WWCode, NYC chapter.

Let's get started.

What goes on a résumé?

Let's start with this question:

What are companies and hiring managers looking for?

Well, they're looking for candidates who:

can improve the company's products with their technical expertise,
be enjoyable to work with, and
can contribute positively to the company's culture and growth.

Even if the recruiter skims through your résumé for less than a minute, you should stand out as a prospective candidate, yes?

For this to happen, the content on your résumé should be:

Recent: Always present information in reverse chronological order—starting with the most recent experience first.
Relevant to the role that you're applying for.
Clear even to a reader who has no context.

Typically, your résumé should only be about a page long. And that's all you've got to make an impression on the reviewer.

As they say, "You're much more than a one-page résumé—but your résumé should not be more than one page.😄

Format of a résumé

There's no one recommended format to draft your résumé. However, the following sections should typically be present:

- Name and Contact info
- Objective (optional)
- Education
- Technical Experience
- Skills
- Leadership | Volunteering

Let's now visit each of these sections, and see how you can best structure each of them.

Name and contact info

✅ Include your name, your email address, links to your portfolio/GitHub.

✅ Be sure to check if your email address is professional enough.

✅ Include social media handles—like LinkedIn—only if you've updated them.

Objective (`optional`)

Include the objective section only if you aim at providing some context to the reviewer.

Every line should tell the recruiter something that they don't already know.

For example, if you're a CS major applying to a software engineering role, your objective isn't going to provide any context to the recruiter.

On the other hand, suppose you're a professional accountant, who's looking to break into software development. Then, the objective tells the recruiter upfront that you're trying to switch careers—and they won't look for a CS degree or developer experience as they skim through your résumé.

Education

Always cite details of your education—starting from your highest qualification first.

If you're a Master's student, only mention details of your Master's and undergraduate degree. Some companies do have a certain cut-off GPA, so be sure to include your GPA.

Some people do include Relevant Coursework subsection in their Education tab.

However, you should use it only if needed.

Being a CS major, doing courses in algorithm design and analysis, and operating systems isn't any interesting to the reviewer. If you're from a non-CS stream, but have supplemented your coursework with courses from the CS bucket—you may include them in the Relevant Coursework section.

Technical Experience

This section should account for nearly 80% of your résumé, and should include:

Relevant work experience and
Projects

▶ We'll talk about this in greater detail in the next section.

Skills

You should always organize your skills by category—ordered by proficiency.

Here's an example:

Languages: Python, JavaScript
Libraries: NumPy, pandas, scikit-learn
Tools: Git

You should always remember to demonstrate your skills in the other sections.

For example, if Python is the language that you're most proficient in—your projects should be indicative of your proficiency.

Leadership | Volunteering

If you've been volunteering, involved in open-source communities, mentoring and the like, you may include them in this section.

Ensure that you're communicating your interests and impact clearly, and keep this section short and towards the end of your résumé.

How to describe your technical experience

This section is the most crucial section in your résumé—be sure to draft this section carefully.

Here are a few suggestions on how you should explain your experience and projects.

❌ Do not list down your job responsibilities.
✔ Write what you accomplished.

❌ Do not tell what you learned.
✔ Instead, explain what you built with that knowledge.

❌ Do not use weak language.

Avoid phrases like:

- Helped build, 
- worked as part of the team,
- helped implement

✔ Use strong language that's impactful.

Say:
- Built,
- Worked on,
- Implemented

❌ Do not be vague when specifying impact.

...worked on speeding up the inference pipeline 
--> # not quantifying impact

✔ Quantify impact wherever possible—talk numbers!.

...worked on speeding up the inference pipeline by 30% 
by reducing the inference time to 2.5 ms --> #quantifying impact

❌ Do not include many projects without explaining each of them.
✔ Explain your projects clearly in detail—prioritize quality over quantity.

Be sure to specify the programming language, and tech stacks used.

Built a ____ using JavaScript, React.

Used Python to code a process scheduler.

Never leave the recruiter guessing why the project is interesting/relevant. Explain clearly.

Now that you know how to draft all major sections in your résumé, let's list down a few concluding points.

Points to remember

Have a résumé for every role that you'd be applying to.

If you're interested in both software engineering and data analytics, be sure to draft a dedicated résumé for each of these roles.

Don't be terse in explaining your projects—including portfolio links doesn't suffice.

The recruiter may not have time to look at your portfolio. So your résumé should do the talking for you.

Do not use fuzzy language just to make your projects sound cool and complicated.

Prefer using simple and clear language instead—just the way you'd explain to your friends.

🎯 And you've reached the end of this post on best practices in résumé writing.

Conclusion

Thanks for reading all the way up to here!😄

Hope you found this post helpful. If you know someone who'd find this useful, do share with them as well.

If you're currently looking for opportunities, I wish you loads of luck in your endeavors.

String Slicing in Python, Explained

Bala Priya C — Mon, 03 Jan 2022 09:35:33 +0000

String slicing lets you slice into Python strings, and work with their slices, or substrings—instead of the whole string.

As strings in Python are immutable, you cannot change them in place. In this tutorial, you'll learn to slice Python strings and work with substrings.

Python String Slicing Syntax

<str>[start:stop:step]

The above line of code:

Returns a slice of the string <str>—starting at index start, extending up to stop-1 in steps of step.
The start index is optional: the slice starts from the beginning of the string by default.
The stop index is also optional: the slice extends up to the end of the string by default.
The step value is optional too. The default value of step is 1 and includes all characters in the string.

Python String Slicing Example

my_str = "Python3"

▶ Let's use enumerate() and examine the characters at each index in the string.

for idx,char in enumerate(my_str):
  print(f"At index {idx}: letter {char}")

# Output
At index 0: letter P
At index 1: letter y
At index 2: letter t
At index 3: letter h
At index 4: letter o
At index 5: letter n
At index 6: letter 3

In the above code, you've used the enumerate() function in conjunction with the for loop. This lets you loop through iterables, and access items along with their indices simultaneously—without having to use the range() function to get the indices.

▶ Let's now use string slicing.

# With `start` and `stop` indices
print(my_str[1:6])
# Output: ython

# Without `stop` index
print(my_str[1:])
# Output: ython3

# Without `start` index
print(my_str[:5])
# Output: Pytho

# With `step = 2`, slice includes every second character
print(my_str[::2])
# Output: Pto3

# Without `start`, `stop` and `step`: slice is entire string
print(my_str[::])
# Output: Python3

Python String Slicing with Negative Step

When you set step to a negative value, you can get slices starting from the end of the string—reverse substrings.

If step = -1 you get a slice starting from the end of the string, and including every character.

This can be super handy when you'd like to reverse a string, like this:

print(my_str[::-1])

# Output: 3nohtyP

Conclusion

To sum up, <str>[start:stop:step] is the syntax to obtain string slices or substrings in Python.

Now that you've learned how to slice strings, it's time to put your skills to practice, maybe? Happy learning and coding!

Cover image: Photo by Tamanna Rumee on Unsplash

A 2021 Reflection Journal: My Tech Writing Journey, Learning, and More

Bala Priya C — Thu, 30 Dec 2021 10:49:40 +0000

With the COVID pandemic bringing about a paradigm shift to virtual, a lot of us got to explore and unleash new opportunities—despite all the odds!

I got into technical content writing roughly a year ago. And 2021 has helped me etch a good growth curve in writing.

Getting started with writing

In late 2020, I had started writing tech tutorials on Medium. All my tutorials were summaries of what I was learning back then. So writing those posts really helped me fill my gaps in understanding, and the feedback from the readers motivated me to keep going.

However, as Medium was a paid platform with articles on all things life, work, and beyond—I was looking to start writing on a free platform, where I'd also get to read more developer-focused content—and get inspired by experienced developers.

In November 2020, I joined the writing team at OpenMined, an open-source organization working in the realm of Privacy-Preserving AI. There, I started learning more about the privacy principles guiding ML and the like. I also started summarizing talks from their Privacy Conference in the form of blog posts for the community.

Thus ended 2020; To new beginnings! Hello 2021🎉✨

Leading the writing team at OpenMined

In January, I got the opportunity to lead the writing team at OpenMined. Leading a team of over 30 writers from over 10 time zones—was a challenging yet rewarding experience.

I led the writing team for a over 6 months, from January through June 2021. During this period, I coordinated efforts to ensure consistent content on the OpenMined blog that members of the community could learn from. And I continued to write a few posts as well.

If you're interested in reading my posts on privacy, and privacy-preserving AI, I've linked to my author profile here

Joining DEV community

I had also started writing on dev.to in early 2021. I liked the fact that I could now read a lot of well-written tutorials on a wide variety of tech stacks and loved how DEV community is run by the devs for the devs.

I spent my first few weeks reading more, and writing less.

I found features like creating a series, being able to include the canonical URL, writing in Markdown, and the fact that all of the content was accessible for free super cool. And I continued writing.😊

I was following writers like Chris Bongers @dailydevtips1 and could only get inspired more!

Writing for freeCodeCamp

I was trying to code my way through freeCodeCamp's Responsive Web Design curriculum, when I got to know that I can apply to become a writer for freeCodeCamp's publication. ✨ I was super happy when Abbey (Abigail Rennemeyer), who is the managing editor onboarded me as a contributor shortly after.

From June to November 2021, I wrote around 18 tutorials—predominantly covering topics in core Python.

You can find all posts that I've written for freeCodeCamp in my author profile. And according to stats from Google Analytics, readers spend 1000+ hours every month reading my beginner-friendly tutorials.

Joining AI Time Journal as an associate editor

I came across another interesting volunteering opportunity at the AI Time Journal while scrolling through my LinkedIn feed. It was regarding an associate editor role with their team. As an associate editor, I had the opportunity to interview guests—who were all thought leaders and experts in data science, and subsequently publish interview posts featuring them.

The guests that I got to interview include startup founders in the data-driven decision making and privacy space, and engineering and program managers at Google, Meta (then, Facebook 😄) and the like.

In case you're interested in data science, you can find all the interviews in this link.

A much-needed break

As a grad student who was and still is trying hard to get past the finish line, I had a burnout. And I decided to stay off the social media radar for a few weeks—deleting a couple of accounts permanently, and checking even school emails sparingly.

This much-needed break indeed helped me get back on track, and regain the lost momentum.

Ending 2021 on a high note

During one of the conversations, @atapas (Tapas Adhikary)—one of the best content creators we have—had suggested that I should also consider writing on Hashnode for there's a vibrant and supportive community around it. And I had made my decision at once.😄

Inspired by great and consistent writers like Tapas and Chris, I decided to join Hashnode. It has been a little over a week, and I love the community already.

And I certainly look forward to a great experience!

I'm super thankful to wonderful communities like OpenMined, freeCodeCamp, dev.to, and Hashnode for having me in 2021. This year has indeed been a great year!

The road ahead!

Though I'm happy with my progress, there's a lot of scope for improvement, and there's a lot I could have done better. But I'm going to be easy on me, and try to get them all right in the coming year. 😄

I'd like to code and write more consistently on Hashnode and dev.to, complete the 100DaysofCode challenge, and help others in their coding journey.

And many congratulations to each of you for staying courageous, and pulling through another difficult year. Do take a moment to reflect on your journey, and be thankful for how far you've come! ✨

For they say, "The more thankful you are for what you have—the more you'll have to be thankful for!"

Thanks again for reading, and have a great year ahead!🎉🎊

PyDP: A Python Differential Privacy Library

Bala Priya C — Mon, 27 Dec 2021 13:12:00 +0000

Outline

1️⃣ What does Differential Privacy try to address?
2️⃣ Why doesn't anonymization suffice?
3️⃣ PyDP example walkthrough

What does Differential Privacy try to address?

Differential privacy aims at addressing the paradox of learning nothing about an individual while learning useful information about a population. In essence, it describes the following promise, made by a data holder, or curator, to a data subject:

“You will not be affected, adversely or otherwise, by allowing your data to be used in any study or analysis, no matter what other studies, data sets, or information sources, are available.” – Cynthia Dwork in The Algorithmic Foundations of Differential Privacy.

Differential Privacy ensures that any sequence of outputs, which are responses to queries is ‘essentially’ equally likely to occur, independent of the presence or absence of any individual’s record.

Consider the illustration below, where the two databases Database#1 and Database#2 differ by only one record, say, your data. If the results obtained from querying the database under these two different settings, are almost the same or similarly distributed, then they essentially are indistinguishable to an adversary.

Illustration of Differentially Private Database Mechanism (Image Source)

Mathematically,

P r [M (d) \in S] \leq e x p (ϵ) P r [M (d') \in S]

where, d and d’ are two subsets of data that differ by a single training example.

M(d) is the output of the training algorithm for the training subset d and M(d’) is the output of the training algorithm for the training subset d’.
The probabilities that these outputs belong to a specific set S under both these conditions should be arbitrarily close. The above equation should hold for all subsets d and d’.

Smaller the value of Ɛ, stronger the privacy guarantees.

Membership Inference Attack (MIA) attempts at determining the presence of a record in a machine learning model’s training data by querying the model. From the discussion above, as the inclusion or exclusion of an individual’s data record cannot be inferred, differential privacy ensures protection against such attacks.

Differentially private database mechanisms can therefore, make confidential data widely available for accurate data analysis.

Why doesn't anonymization suffice?

The Netflix Prize was an open competition for the best collaborative filtering algorithm for movie recommendations.

The dataset released was anonymized, without the users or the films being identified except by numbers assigned for the contest. Such anonymized movie records were published by Netflix as training data for the competition.

However, there were several users who could be identified by linkage with the Internet Movie Database (IMDb) which was non-anonymized and publicly available.

Researchers Arvind Narayanan and Vitaly Shmatikov, at the University of Texas at Austin present their studies in their work Robust De-anonymization of Large Datasets (How to Break Anonymity of the Netflix Prize Dataset).

Therefore, such linkage attacks can be used to match “anonymized” records with non-anonymized records in a different dataset.

Differential privacy aims at neutralizing such linkage attacks.
As Differential Privacy is a property of the data access mechanism, it is unrelated to the presence or absence of auxiliary information available to the adversary.

Therefore, access to the IMDb would no longer permit a linkage attack to someone whose history is in the Netflix training set than to someone not in the training set.

De-anonymization of users in the Netflix Prize contest (Image Credit: Arvind Narayanan)

PyDP Example Walkthrough

PyDP is OpenMined’s Python wrapper for Google’s Differential Privacy project. The library provides a set of ε-differentially private algorithms, which can be used to produce aggregate statistics over numeric datasets containing private or potentially sensitive information.

Installing PyDP

!pip install python-dp # installing PyDP

Necessary Imports

import pydp as dp # by convention our package is to be imported as dp (dp for Differential Privacy!)
from pydp.algorithms.laplacian import BoundedSum, BoundedMean, Count, Max
import pandas as pd
import statistics 
import numpy as np
import matplotlib.pyplot as plt

Fetch all the required data!

The dataset used here contains 5000 records, and is stored across 5 files, each file containing 1000 records.

More specifically, the dataset contains details such as the first and last names, email addresses of customers and the amount they spent on purchasing goods, and the state in the US they're from.

Let's fetch all the records, read them into pandas DataFrames and take a look at the head of each of the DataFrames.

url1 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/01.csv'
df1 = pd.read_csv(url1,sep=",", engine = "python")
print(df1.head())

url2 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/02.csv'
df2 = pd.read_csv(url2,sep=",", engine = "python")
print(df2.head())

url3 ='https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/03.csv'
df3 = pd.read_csv(url3,sep=",", engine = "python")
df3.head()

url4 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/04.csv'
df4 = pd.read_csv(url4,sep=",", engine = "python")
print(df4.head())

url5 = 'https://raw.githubusercontent.com/OpenMined/PyDP/dev/examples/Tutorial_4-Launch_demo/data/05.csv'
df5 = pd.read_csv(url5,sep=",", engine = "python")
print(df5.head())

Now that we've fetched records from all the 5 files, let us concatenate all the DataFrames into a single large DataFrame and this constitutes our original dataset. Note that our dataset has 5000 rows(records) and 6 columns.

combined_df_temp = [df1, df2, df3, df4, df5]
original_dataset = pd.concat(combined_df_temp)
print(original_dataset.shape)

# Result
# (5000,6)

Creating a Parallel Database

Let us now create a parallel database that differs by only one record, say, Osbourne's record and name it redact_dataset. We then inspect the heads of both DataFrames to verify that Osbourne's record has been removed.

redact_dataset = original_dataset.copy()
redact_dataset = redact_dataset[1:]
print(original_dataset.head())
print(redact_dataset.head())

At this point, let us ask ourselves the following question.

Is the amount of money we spend at our neighborhood store private or sensitive information? Well, it may not seem all that sensitive! But, what if the same information can be used to identify us?

In the example that we have, let us say we remove all personal information such as name and email address. Given that there's some access to the store's sales record, will the sales amount in itself not suffice to infer Osbourne's identity? Yes!

And to do that, we sum up all entries in the sales_amount column in our original dataset, and the redact_dataset. The difference between these two sums exactly gives us the amount that Osbourne spent, and is verified as shown in the code snippet below.

This is a simple example where membership inference was successful even after removal of personally identifiable information.

sum_original_dataset = round(sum(original_dataset['sales_amount'].to_list()), 2)
sum_redact_dataset = round(sum(redact_dataset['sales_amount'].to_list()), 2)
sales_amount_Osbourne = round((sum_original_dataset - sum_redact_dataset), 2)
assert sales_amount_Osbourne == original_dataset.iloc[0, 4]

Differentially Private Sum

Now, we illustrate how differentially private sum in place of simple sum can help in rendering membership inference attacks unsuccessful.

For the example above, let's assume that the customers should spend a minimum of 5$ at the store and no more than 250$ for a particular purchase.

We then go ahead and compute differentially private sum on both original and the parallel dataset that differed by one record, as shown in the code snippets below.

dp_sum_original_dataset = BoundedSum(epsilon= 1.5, lower_bound =  5, upper_bound = 250, dtype ='float') 
dp_sum_og = dp_sum_original_dataset.quick_result(original_dataset['sales_amount'].to_list())
dp_sum_og = round(dp_sum_og, 2)
print(dp_sum_og)

# Output dp_sum_og
# 636723.61
dp_redact_dataset = BoundedSum(epsilon= 1.5, lower_bound =  5, upper_bound = 250, dtype ='float')
dp_redact_dataset.add_entries(redact_dataset['sales_amount'].to_list())
dp_sum_redact=round(dp_redact_dataset.result(), 2)
print(dp_sum_redact)

# Output dp_sum_redact
# 636659.17

Let's proceed to summarize a few observations.

Now that we've calculated the differentially private sum on the original and the second dataset, it's straightforward to verify that that the differentially private sums are not equal to sums under the non-differentially private setting.
Also, the difference is no longer equal to the amount that Osbourne spent indicating that membership attacks would now be unsuccessful, regardless of access to any other customer records.
Interestingly, the differentially private sum values are still comparable and are not very different.

We've therefore succeeded in ensuring differential privacy in our simple example!

print(f"Sum of sales_value in the orignal dataset: {sum_original_dataset}")
print(f"Sum of sales_value in the orignal dataset with DP: {dp_sum_og}")
assert dp_sum_og != sum_original_dataset

# Output
Sum of sales_value in the orignal dataset: 636594.59
Sum of sales_value in the orignal dataset with DP: 636723.61

print(f"Sum of sales_value in the second dataset: {sum_redact_dataset}")
print(f"Sum of sales_value in the second dataset with DP: {dp_sum_redact}")
assert dp_sum_redact != sum_redact_dataset

# Output
Sum of sales_value in the second dataset: 636562.65
Sum of sales_value in the second dataset with DP: 636659.17

print(f"Difference in Sum with DP: {round(dp_sum_og - dp_sum_redact, 2)}")
print(f"Actual Difference in Sum: {sales_amount_Osbourne}")
assert round(dp_sum_og - dp_sum_redact, 2) != sales_amount_Osbourne

# Output
Difference in sum using DP: 64.44
Actual Value: 31.94

Hope this introductory post helped in understanding the intuition behind differential privacy and protection against membership inference attacks. We shall look at a few more examples in subsequent blog posts.😊

References

[1] The Algorithmic Foundations of Differential Privacy by Cynthia Dwork.

[2] PyDP Tutorial by Chinmay Shah at OpenMined Privacy Conference

Private AI: Machine Learning on Encrypted Data

Bala Priya C — Fri, 24 Dec 2021 13:25:03 +0000

Kristin Lauter is currently the head of research science at Facebook AI Research (Meta), formerly the head of Cryptography and Privacy Research Group at Microsoft Research. This post is based on her talk at the OpenMined Privacy Conference.

What is the privacy problem with AI?

Let us begin by looking at a generic Machine Learning (ML) algorithm that takes in our data as input and outputs some kind of decision - a classification label, numerical value or a recommendation.

The privacy problem stems from the fact that we have to input our data to get those nice and valuable predictions.

Many AI applications powered by smart agents are hosted on the cloud, and protecting privacy of user data is pivotal to building secure applications.

The privacy of data can be protected through encryption. However, standard encryption methods do not allow computation on encrypted data. Here's where Homomorphic Encryption (HE) helps.

What is Homomorphic Encryption?

In simple terms, Homomorphic Encryption is a mathematical tool:

that allows for encryption of data,
ensuring privacy while at the same time, allowing computations to be performed on the encrypted data.
The result of computation can then be decrypted to get the results.

As shown in the figure below, with Homomorphic Encryption (HE), the order of encryption and computation can be interchanged.

Suppose you have data a and b. You can perform computation on the data and then encrypt the result, denoted by E(a*b).

Alternatively, you could encrypt the data and then perform computation, denoted by E(a) * E(b) in the figure below. If the encryption is homomorphic, then these two values, E(a*b) and E(a) * E(b) decrypt to the same value.

Therefore, we can choose to encrypt private data a and b,
outsource computation, say, to the cloud, and decrypt the obtained result of the computation to view the actual meaningful results.

Understanding Homomorphic Encryption intuitively through Homer-morphic Encryption

Let's try to think of a fictional character and draw a relatable analogy.

Remember Homer Simpson from 'The Simpsons'?🙂

The following illustration is aimed at giving an intuitive explanation to what Homomorphic Encryption does.

Let us say you need to get a jewel made and you have your gold ready! You'd now like to call your jeweler (Homer Simpson) and get your jewel made, however you're not very sure if your jeweler is trustworthy. Here's a suggestion.

You may place your gold in a box, lock it and keep the key with yourself. You may now invite over your jeweler and ask him to work on the gold nuggets through the glove box. Once the jewel is done, you may unlock your box and retrieve your jewel. Isn't that cool?

Let us try to parse the analogy a bit.

Your private data is analogous to gold,
outsourcing computations on encrypted data in a public environment is similar to getting your jeweler to work on the gold through glove box, and
decrypting the results of computation to view the meaningful results is analogous to opening the box to get your jewel after the jeweler has left. 🙂

Without delving deep into the math involved, the high-level idea behind homomorphic encryption is as follows.

Homomorphic Encryption uses lattice-based encoding.
Encryption adds noise to a secret inner product.
Decryption subtracts the secret inner product and the noise becomes easy to cancel.

In the next section, let's see the capabilities of Microsoft SEAL, a library for Homomorphic Encryption.

Microsoft SEAL (Simple Encrypted Arithmetic Library)

SEAL (Simple Encrypted Arithmetic Library) is Microsoft's library for Homomorphic Encryption (HE), widely adopted by research teams worldwide. It was first released publicly in 2015, followed by the standardization of HE in November 2018. Microsoft SEAL is available for download and use here.

In recent years, availability of hardware accelerators has also enabled several orders of magnitude speedup. Here's a timeline of how Homomorphic Encryption has been adopted due to advances in research and easier access to compute.

Idea: Computation on encrypted data without decrypting it
2009: Considered impractical due to substantial overhead involved
2011: Breakthrough at Microsoft Research
Subsequent years of research: Practical encoding techniques that achieved several orders of speed-up
2016: Crypto Nets at ICML 2016 - Neural Net predictions on encrypted data

Now that we're familiar with how HE has been adopted over the years, the subsequent sections will focus on the possible applications of HE.

Cloud Compute Scenarios benefitting from HE

The following are some of the cloud computing scenarios that could potentially benefit from Homomorphic Encryption (HE):

Private Storage and Computation
Private Prediction Services
Hosted Private Training
Private Set Intersection
Secure Collaborative Computation

The markets benefitting from such services include healthcare, pharmaceutical, financial, government, insurance, manufacturing, oil and gas sector, to name a few. A few applications of Private AI across industries are listed below.

Finance: Fraud detection, automated claims processing, threat intelligence, and data analytics are some applications.
Healthcare: The scope of Private AI in healthcare includes medical diagnosis, medical support systems such as healthcare bots, preventive care and data analytics.
Manufacturing: Predictive maintenance and data analytics on potentially sensitive data.

Azure ML Private Prediction

An image classification model for encrypted inferencing in Azure Container Instance (ACI), built on Microsoft SEAL, was announced at Microsoft Build Conference 2020. The tutorial can be accessed here.

References

[1] Recording of the PriCon talk

How AI Has Changed Over the Years: Hear from Denis Rothman, AI Expert

Bala Priya C — Thu, 23 Dec 2021 14:03:37 +0000

Earlier this year, I had the opportunity to interview Denis Rothman, an AI expert and a best-selling author at the AI Time Journal.

Denis Rothman is the author of several popular books such as Transformers in Natural Language Processing, and Hands-On Explainable AI.

In this interview, Denis Rothman shares his academic background and research interests. He also shares valuable insights stemming from his decades of experience in the field of AI, including:

Academic Background and Specialization
Changes in the AI Landscape over the years
Advances in AI Research
Journey to Becoming a Best-Selling Author
Importance of Writing in the Learning Process

Academic Background and Specialization

🎙️Bala: Could you tell us about your academic background and specialization?

Denis Rothman: In the 1970s and 1980s, I studied Linguistics, Civilizations, and Computer Science at Sorbonne University and Paris-Diderot University. When I graduated, I taught Computer Science at Pathéon Sorbonne. I started my freelance Linguistics and AI activity while I was a student. During that period, I registered a “word2vector” patent that gained university (post-graduate degree), media, and corporate recognition. I obtained public funding to continue my research. In 1986, I registered what is now a “chatbot” patent that led me directly to LVMH (Moët et Chandon division) and Airbus (formerly Aerospatiale division).

My professors were cross-disciplinary and encouraged exploration. I traveled to many countries to understand how civilizations were built, how languages were structured.

I became familiar with Supply Chain Management (SCM) which 
led directly to AI SCM projects. I rapidly specialized 
in Artificial Intelligence through a passion for algorithms.

Changes in the AI Landscape over the years

🎙️Bala: As an expert in the field of AI, could you share your insights on how the AI landscape has changed over the years?

Denis Rothman: In the 1980s, we worked on personal computers or small mainframes (servers). We did not have the Internet, so we had to physically go around the world to meet people and pick up new ideas. This explains why I traveled so much during my college and early freelance years.

In the 1980s and 1990s, you had to have a solid reputation for the word to go around to deliver AI optimizing algorithms for Supply Chain Management successfully. I focused on optimizing, so I made it fine during those years.

Starting in 2006, cloud servers became available, and the Internet reached a good level of maturity. It changed everything! No more traveling so much, no more installations. We could sell SaaS (Software as a Service). The users would have to connect with their browsers, test for free then pay a monthly license! On top of that, we could access good papers and documentation from around the world. We could also share our ideas.

When artificial intelligence became a mainstream trend 
around 2015, we were ready, had the solution and the 
experience. So, a fantastic journey began and is still continuing!

Advances in AI Research

🎙️Bala: What, according to you, are the most significant advances in AI research in the recent years?

Denis Rothman: The following three trends are mind-blowing:

The arrival of powerful Artificial Intelligence models trained with supercomputers such as OpenAI’s GPT-3 engines. These models are industrialized using structured transformer models, for example. The texts they generate cannot be distinguished from human expression.
The embedding of Artificial Intelligence everywhere, from our smartphones to our social media activity. Google Search has improved tremendously provided snippets of the search answer without clicking on a link, for example. Google Translate is progressing daily. Everything in AI is evolving very fast!
Industry 4.0 Artificial Intelligence, which is used to connect everything to everything, everywhere. So from China to the US, from Europe to Africa, the world is being connected physically and digitally from the Americas to everywhere!

🎙️Bala: What do you think are the challenges in adopting AI at scale in the industry?

Denis Rothman: There are two key factors beyond the classical project management constraints encountered during any project:

1. Mindset

It is challenging to explain that Artificial Intelligence is only a tool. It’s not organic. It is not a living organism.

AI is only math. 
You can either be the tool of AI 
or use it as a tool.

It is also tough to explain that Industry 4.0 is generated millions of micro-decisions in the supply chain flows around the world. Without AI, there would not be enough humans to face these challenges without slowing the world down slower food delivery, vaccine production, clothing, and everything we need daily.

2. Cross-disciplinary knowledge

To be able to fully understand the requirements of a project, cross-disciplinary knowledge is mandatory. For example, Linguistics for NLP or notions of law for applications of AI in governance and other legal applications.

🎙️Bala: According to you, what are the areas in AI that offer some of the most exciting research opportunities?

Denis Rothman: I love Artificial Intelligence because I love algorithms. The most interesting aspect of any AI project is creating algorithms. Programming is another exercise. Programming is translating an algorithm into code. Programming comes with constraints, bugs, machine power, and criticism from others.

But when you are designing an algorithm in the middle of a quiet
night, it’s like communicating with the structure of the universe. 

It is like writing a music score, drawing, or painting. 

You are creating something out of nothing but yourself!

Computer Vision, Natural Language Processing(NLP), and all 
of the AI-ML-DL algorithms are fascinating to design! 
It’s like playing different music scores.

Journey to Becoming a Best-Selling Author

🎙️Bala: As an author of several books on popular topics in AI, such as Transformers, Explainable AI, could you please tell us about your journey as a technical author?

Denis Rothman: I love Artificial Intelligence, people, and sharing. I began writing about ideas, logic, and reasoning when I was in high school. Sharing has always been exhilarating for me. When I share an idea, the person will react and ask difficult questions. In turn, this forces me to think harder and explain better. After my ideas are structured, we both see the light together!

Sharing is enjoying togetherness in the society. 
You do something, you analyze the reaction, and you adapt.

If you do not share, you do not learn.
If you do not share, you cannot understand what you are doing because you need a third party to tell you what they see in your ideas beyond your confined ecosphere.

I wrote a lot of documentation for corporations and some studies. However, in 2017, Tushar Gupta, a visionary working at Packt Publishing, encouraged me to share my ideas, knowledge, and experience through books. Packt has a unique ability to create publishing teams. You are never alone. So you can share your work and get a lot of feedback before anything is published. That is fantastic!

Importance of Writing in the Learning Process

🎙️Bala: What is the importance of writing in the process of learning AI? I strongly believe that writing reinforces understanding and the learning process. What’s your opinion on writing tech tutorials when learning a particular concept?

Denis Rothman: There are basically 4 approaches to language:

1. Understanding through sound

The words cross your mind at full speed. It’s difficult not to miss a lot of content. You can get information through prosody (intonations and tone of voice, for example). However, in the world of technology, content matters more than intonation.

2. Speaking through sound

When you speak, you are compelled to be clear, or nobody will understand you. However, you don’t have to worry about punctuation, the length of sentences. Also, unless you are reading a text, you are going at full speed without controlling the content.

3. Reading

Reading is no doubt deeper than listening. You can capture quite a lot of information and think about it for as long as you wish. You are picking up a lot of ideas.

4. Writing

Writing is the ultimate form of communication. If you did your research correctly, it’s the sum of the three previous dimensions!

First, you have listened to others.
Second, you have made an effort to explain yourself through speech.
Third, you have picked up a lot of information by reading others.

Now, you can compile everything you feel and know into the content you can share! You have prepared quite a work of art for your readers. You are now eager to know what they think and start a communication process at another level.

🎙️Bala: What are some of the challenges associated with using AI in Healthcare?

Denis Rothman: Privacy in Healthcare worries me. We are not machines, robots, or things. We are human beings! We deserve respect and the ability to keep our dignity. We are not lab rats.

That being said, medical progress relies on statistics! 
Using AI to find patterns is undoubtedly necessary.

I think that government-led ethical committees should decide what is good or bad for us, not private sector investors. That is about all I can say as a citizen because I’m not a Healthcare expert! 🙂

🎙️Bala: If you were to suggest a few books that can help AI enthusiasts to gain strong foundational skills, what would they be?

Denis Rothman: The tempting answer is “mine”. 🙂

I’m going to disappoint you because I do not believe in flashy knowledge. Artificial Intelligence is there to help machines make decisions for us about our culture. To design effective AI, we need to be educated.

First, I would recommend learning to review Mathematics in detail: Arithmetic, General Algebra, Calculus, Geometry, Trigonometry. I would then go in this order among many others: Plato, Aristotle, Descartes, Pascal, Kant, Nietsche, Levi-Strauss, Boltzmann, Poincaré, Markov, Lyapunov, Einstein, Shannon, and Turing. Then, the army of AI authors, yours truly included!

What Is Static Code Analysis?

Bala Priya C — Mon, 06 Dec 2021 17:14:56 +0000

Working on a large software project can often be a challenging yet rewarding experience. As a developer, you write code to solve problems.

However, you’ll have to write secure, bug-free code, and follow best coding practices while complying with standards. And this can be super difficult when you’re working under pressure.

This is where Static Code Analysis can come in handy.

By the end of this post, you’ll have learned enough to answer the following questions:

What is static code analysis?
Why should you care about static analysis as a developer?
What are the advantages and limitations of static code analysis?

Static Code Analysis

In simple terms, static analysis, or static code analysis scans your code to identify potential bugs, weaknesses, and anti-patterns—all without actually executing the code.

So how exactly does static analysis work under the hood?

Well, static analysis often involves parsing of the source code into an intermediary representation on which you can run analysis. This analysis then returns potential security issues, bugs, and performance issues in your code—which you can fix almost immediately.

Now that you know what static analysis is, you’ll learn how it can help you write better code in the next section.

Why Should You Care About Static Analysis as a Developer?

As a developer, static analysis helps you in every step of the development process. You can:

write code,
run static analysis,
review, and
fix reported bugs—without having to wait for code reviews.

And it’s a lot easier to fix bugs as and when they arise rather than having to regress and do it much later.

Unlike the conventional testing process, where you need a specific module to be all ready to go, static analysis only requires a chunk of code that can be modeled.

The term ‘modeled’ here means that the code can be parsed into that intermediary representation on which the analyzer can be run. Please note that your code is never actually run in the process.

This intermediary representation is often the Abstract Syntax Tree (AST).

If you'd like to know more about ASTs, consider giving this post a read.

Abstract Syntax Tree (AST) - Explained in Plain English

Bala Priya C ・ Dec 6 '21

#programming #codequality #compilers #codereview

Because of how static analysis can be integrated into development right from the beginning, you can run analysis at every step and make sure your code meets the standards.

This ensures that your code is almost bug-free when the actual testing process begins.

Advantages of Static Analysis

So far, you’ve learned how you can benefit from static analysis as a developer. Let’s enumerate its advantages:

Static analysis is much faster and a lot more efficient than manual code reviews.
The earlier you detect bugs and security issues, the easier it is to fix them. This saves time and substantially reduces cost.
Waiting until deployment and then checking for vulnerabilities, regressing, and making fixes is needlessly complex and time-consuming.
By shipping software that’s backed by quality code, you build trust, gain and retain customer base. This in turn motivates you to build more helpful products.

Limitations of Static Analysis

Well, are there any caveats yet? Let’s go over the limitations of static analysis:

On the flip side, static analysis is prone to false positives. The analysis may report too many bugs, some of which may not actually be bugs.
For a large codebase, reviewing each of the reported bugs and vulnerabilities could come across as overkill. Therefore, running static analysis as you code your way through the project is recommended.
Static analysis can miss certain subtle bugs which require human expertise. But it’s possible to fix them in subsequent steps of code review.
And it’s always possible to improve the static analysis tool by embedding this intelligence to identify new bugs—therefore, static analysis is extensible to find new bugs.

Hope this post helped you get a high-level overview of static analysis, its advantages and limitations. See you all soon in another post!

DEV Community: Bala Priya C

Descriptive Statistics with Python for Beginner Data Scientists

Prerequisites

Table of Contents

What Is Descriptive Statistics?

The Dataset

Measures of Central Tendency

Mean

Median

Visualizing Mean vs Median

Measures of Spread

Standard Deviation

Variance

Range and IQR

Visualizing Spread: The Box Plot

The Five-Number Summary

Skewness

Visualizing Skewness: The Histogram

Putting It All Together

Which Chart for Which Job?

Summary

What's Next?

How to Work with Parquet Files in Python – A Guide with Examples

Table of Contents

What Is the Parquet File Format?

Prerequisites

Writing Parquet Files in Python

Reading Parquet Files in Python

Reading Specific Columns

Writing Parquet Files with Compression

Working with Row Groups and Filtering

A Practical Example: Analyzing Sales Data

When Should You Use Parquet?

Conclusion

5 DIY Python Decorators for Building Cleaner Data Pipelines

Prerequisites

How Python Decorators Work

1. @retry — Handle Transient Failures

2. @timer — Measure Execution Time

3. @validate_schema — Catch Bad Data Early

4. @cache_result — Skip Redundant Computation

5. @log_step — Build a Trail

Wrapping Up

MIT's Missing Semester Class: Beyond the CS Curriculum

So What Does This Course Teach?

#1 – Shell and Shell Scripting

#2 – Working with Vim

#3 – Data Wrangling

#4 – Command-Line Environment

#5 – Version Control with Git

#6 – Debugging and Profiling

#7 – Build Systems, Dependency Management, and More

#8 – Security and Cryptography

#9 – Potpourri

Wrapping Up

How To Code a Simple Number Guessing Game in Python

How Does the Number Guessing Game Work?

Coding the Number Guessing Game

Step 1 - Import the random module

Step 2 - Set up the range and the maximum number of attempts

Step 3 - Generate a random number

Step 4 - Read in the user's input

Step 5 - Validate the user's guess

Step 6 - Track the number of attempts and detect end-of-game conditions

Step 7 - Play the game!

Putting it all together

Wrapping Up

How To Write an Effective Technical Résumé

What goes on a résumé?

Format of a résumé

Name and contact info

Objective (optional)

Education

Technical Experience

Skills

Leadership | Volunteering

How to describe your technical experience

Points to remember

Conclusion

String Slicing in Python, Explained

1. `@retry` — Handle Transient Failures

2. `@timer` — Measure Execution Time

3. `@validate_schema` — Catch Bad Data Early

4. `@cache_result` — Skip Redundant Computation

5. `@log_step` — Build a Trail

Step 1 - Import the `random` module

Objective (`optional`)